{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ClinVar SNV and Non-SNV Processing Pipeline\n",
    "\n",
    "This notebook processes ClinVar genetic variants to create machine learning datasets for variant effect prediction. See `Clinvar_SNV_Non_SNV_README.md` for detailed documentation.\n",
    "\n",
    "## Quick Start\n",
    "\n",
    "1. Update file paths in the configuration section\n",
    "2. Ensure all dependencies are installed\n",
    "3. Run cells in order\n",
    "4. Monitor progress and memory usage\n",
    "\n",
    "**⚠️ Important**: This pipeline requires significant computational resources and storage space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration\n",
    "\n",
    "Update these paths for your environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configuration - Update these paths for your environment\n",
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "# File paths (update these for your system)\n",
    "CONFIG = {\n",
    "    # Input data\n",
    "    'clinvar_vcf': 'data/clinvar_grch38.vcf.gz',\n",
    "    'reference_genome': 'data/reference/GRCh38.fa',\n",
    "    'hgnc_mapping': 'data/hgnc_complete_set.txt',\n",
    "    \n",
    "    # VEP configuration\n",
    "    'vep_root': '/path/to/vep',\n",
    "    'vep_cache': '/path/to/vep/cache',\n",
    "    \n",
    "    # Output paths\n",
    "    'output_dir': 'output',\n",
    "    'temp_dir': 'temp',\n",
    "    \n",
    "    # Processing parameters\n",
    "    'window_size': 4096,\n",
    "    'max_variant_size': 64,\n",
    "    'num_threads': 8,\n",
    "    'batch_size': 100000\n",
    "}\n",
    "\n",
    "SCRATCH_DIR = '/your/scratch/directory'  # Update this to your scratch directory\n",
    "\n",
    "# Create output directories\n",
    "for dir_path in [CONFIG['output_dir'], CONFIG['temp_dir']]:\n",
    "    os.makedirs(dir_path, exist_ok=True)\n",
    "    \n",
    "print(\"Configuration loaded. Please verify all paths are correct:\")\n",
    "for key, value in CONFIG.items():\n",
    "    if 'path' in key or 'dir' in key:\n",
    "        exists = os.path.exists(value) if not key.endswith('dir') else True\n",
    "        status = \"✅\" if exists else \"❌\"\n",
    "        print(f\"  {status} {key}: {value}\")\n",
    "        \n",
    "print(\"\\n📝 Update CONFIG dictionary above with your actual file paths\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ClinVar SNV and Non-SNV Variant Processing Pipeline\n",
    "\n",
    "This notebook processes ClinVar genetic variants (both SNVs and non-SNVs) to create a comprehensive machine learning dataset for variant effect prediction. The pipeline includes:\n",
    "\n",
    "## Overview\n",
    "\n",
    "1. **Data Processing**: Download and process ClinVar VCF data using VEP (Variant Effect Predictor)\n",
    "2. **Sequence Window Extraction**: Generate 4096bp genomic windows centered on variants\n",
    "3. **Feature Engineering**: Extract pathogenicity, disease associations, and gene information\n",
    "4. **Dataset Creation**: Build training/test datasets with disjoint disease splits\n",
    "5. **Quality Control**: Comprehensive statistics and validation\n",
    "\n",
    "## Key Features\n",
    "\n",
    "- **Genomic Windows**: 4096bp sequences with centered mutations\n",
    "- **Variant Types**: Both SNVs and structural variants (insertions, deletions, etc.)\n",
    "- **Clinical Annotations**: Pathogenicity classification and disease associations\n",
    "- **Gene Mapping**: Integration with HGNC gene nomenclature\n",
    "- **Disjoint Splits**: Train/test splits ensuring no disease overlap\n",
    "\n",
    "## Requirements\n",
    "\n",
    "- **Computational Resources**: High-memory system (recommended for large datasets)\n",
    "- **Software Dependencies**: VEP, Python libraries (pandas, pysam, pyarrow, hgvs)\n",
    "- **Reference Data**: GRCh38 genome assembly, HGNC gene mapping\n",
    "- **Storage**: Sufficient space for intermediate files (~100GB+)\n",
    "\n",
    "## Output\n",
    "\n",
    "Final datasets suitable for:\n",
    "- Variant effect prediction models\n",
    "- Pathogenicity classification\n",
    "- Disease association studies\n",
    "- Genomic language model training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial Setup (For HPC/Cluster Environments)\n",
    "\n",
    "**Note**: This section contains setup instructions for high-performance computing environments. Adapt paths and module loading commands for your specific system.\n",
    "\n",
    "### Prerequisites Installation\n",
    "If running on a cluster, you may need to download Python wheels and reference data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download required Python packages and reference data\n",
    "# Adjust paths and module loading for your specific environment\n",
    "\n",
    "# Example for cluster environments:\n",
    "# module load python gcc arrow postgresql\n",
    "\n",
    "# Create directory for Python wheels (adjust path as needed)\n",
    "# mkdir -p /path/to/your/pywheels\n",
    "# pip download hgvs -d /path/to/your/pywheels\n",
    "\n",
    "# Download HGNC gene mapping data\n",
    "# wget -O hgnc_complete_set.txt \"https://storage.googleapis.com/public-download-files/hgnc/tsv/tsv/hgnc_complete_set.txt\"\n",
    "\n",
    "print(\"Setup instructions provided above. Adjust paths for your environment.\")\n",
    "print(\"Required data:\")\n",
    "print(\"- HGNC complete gene set\")\n",
    "print(\"- Python packages: hgvs, pandas, pyarrow, pysam, tqdm\")\n",
    "print(\"- VEP installation with cache\")\n",
    "print(\"- GRCh38 reference genome\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment Setup\n",
    "\n",
    "**For cluster/HPC environments**: Configure virtual environment and load required modules.\n",
    "**For local environments**: Ensure all dependencies are installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Environment setup for cluster/HPC systems\n",
    "# Adjust module loading and paths for your specific environment\n",
    "\n",
    "# Example cluster setup:\n",
    "\"\"\"\n",
    "# Create virtual environment\n",
    "python -m venv /tmp/clinvar_env\n",
    "\n",
    "# Load required modules (adjust for your system)\n",
    "module load python gcc arrow postgresql\n",
    "module load perl samtools tabix bcftools mariadb\n",
    "\n",
    "# Activate virtual environment\n",
    "source /tmp/clinvar_env/bin/activate\n",
    "\n",
    "# Install packages\n",
    "pip install notebook pandas pyarrow pysam hgvs tqdm networkx\n",
    "\n",
    "# Start Jupyter (for remote access)\n",
    "jupyter notebook --no-browser --ip=$(hostname -f) --port=8888\n",
    "\"\"\"\n",
    "\n",
    "# For local environments, ensure these packages are installed:\n",
    "required_packages = [\n",
    "    'pandas>=1.3.0',\n",
    "    'pyarrow>=5.0.0', \n",
    "    'pysam>=0.19.0',\n",
    "    'hgvs>=1.5.0',\n",
    "    'tqdm>=4.60.0',\n",
    "    'networkx>=2.6.0'\n",
    "]\n",
    "\n",
    "print(\"Required packages:\")\n",
    "for pkg in required_packages:\n",
    "    print(f\"  - {pkg}\")\n",
    "    \n",
    "print(\"\\nFor VEP processing, also required:\")\n",
    "print(\"  - VEP (Ensembl Variant Effect Predictor)\")\n",
    "print(\"  - BCFtools, SAMtools, Tabix\")\n",
    "print(\"  - Reference genome and VEP cache files\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/localscratch/naimerja.43836119.0/clinvar_env/bin/python\n"
     ]
    }
   ],
   "source": [
    "!which python\n",
    "# Verify Python environment and core dependencies\n",
    "import sys\n",
    "import subprocess\n",
    "\n",
    "print(f\"Python executable: {sys.executable}\")\n",
    "print(f\"Python version: {sys.version}\")\n",
    "\n",
    "# Check for required packages\n",
    "try:\n",
    "    import pandas as pd\n",
    "    import pyarrow as pa\n",
    "    import pysam\n",
    "    import hgvs\n",
    "    import tqdm\n",
    "    import networkx as nx\n",
    "    \n",
    "    print(\"\\n✅ Core dependencies available:\")\n",
    "    print(f\"  - pandas: {pd.__version__}\")\n",
    "    print(f\"  - pyarrow: {pa.__version__}\")\n",
    "    print(f\"  - pysam: {pysam.__version__}\")\n",
    "    print(f\"  - hgvs: {hgvs.__version__}\")\n",
    "    print(f\"  - networkx: {nx.__version__}\")\n",
    "    \n",
    "except ImportError as e:\n",
    "    print(f\"❌ Missing dependency: {e}\")\n",
    "    print(\"Please install required packages first\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install required packages\n",
    "# Adjust installation method based on your environment\n",
    "\n",
    "# For environments with pre-downloaded wheels:\n",
    "# !pip install --no-index --find-links /path/to/pywheels hgvs\n",
    "# !pip install --no-index tqdm pandas pyarrow\n",
    "\n",
    "# For standard environments:\n",
    "# !pip install hgvs tqdm pandas pyarrow pysam networkx\n",
    "\n",
    "print(\"Package installation commands provided above.\")\n",
    "print(\"Choose the appropriate method for your environment:\")\n",
    "print(\"  - Standard: pip install <package>\")\n",
    "print(\"  - Offline: pip install --no-index --find-links <wheel-dir> <package>\")\n",
    "print(\"  - Conda: conda install <package>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View possible fields from clinvar\n",
    "\n",
    "## ClinVar VCF Data Exploration\n",
    "\n",
    "Examine the structure and metadata of the ClinVar VCF file to understand available annotations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "##fileformat=VCFv4.1\n",
      "##FILTER=<ID=PASS,Description=\"All filters passed\">\n",
      "##fileDate=2025-04-29\n",
      "##source=ClinVar\n",
      "##reference=GRCh38\n",
      "##ID=<Description=\"ClinVar Variation ID\">\n",
      "##INFO=<ID=AF_ESP,Number=1,Type=Float,Description=\"allele frequencies from GO-ESP\">\n",
      "##INFO=<ID=AF_EXAC,Number=1,Type=Float,Description=\"allele frequencies from ExAC\">\n",
      "##INFO=<ID=AF_TGP,Number=1,Type=Float,Description=\"allele frequencies from TGP\">\n",
      "##INFO=<ID=ALLELEID,Number=1,Type=Integer,Description=\"the ClinVar Allele ID\">\n",
      "##INFO=<ID=CLNDN,Number=.,Type=String,Description=\"ClinVar's preferred disease name for the concept specified by disease identifiers in CLNDISDB\">\n",
      "##INFO=<ID=CLNDNINCL,Number=.,Type=String,Description=\"For included Variant : ClinVar's preferred disease name for the concept specified by disease identifiers in CLNDISDB\">\n",
      "##INFO=<ID=CLNDISDB,Number=.,Type=String,Description=\"Tag-value pairs of disease database name and identifier submitted for germline classifications, e.g. OMIM:NNNNNN\">\n",
      "##INFO=<ID=CLNDISDBINCL,Number=.,Type=String,Description=\"For included Variant: Tag-value pairs of disease database name and identifier for germline classifications, e.g. OMIM:NNNNNN\">\n",
      "##INFO=<ID=CLNHGVS,Number=.,Type=String,Description=\"Top-level (primary assembly, alt, or patch) HGVS expression.\">\n",
      "##INFO=<ID=CLNREVSTAT,Number=.,Type=String,Description=\"ClinVar review status of germline classification for the Variation ID\">\n",
      "##INFO=<ID=CLNSIG,Number=.,Type=String,Description=\"Aggregate germline classification for this single variant; multiple values are separated by a vertical bar\">\n",
      "##INFO=<ID=CLNSIGCONF,Number=.,Type=String,Description=\"Conflicting germline classification for this single variant; multiple values are separated by a vertical bar\">\n",
      "##INFO=<ID=CLNSIGINCL,Number=.,Type=String,Description=\"Germline classification for a haplotype or genotype that includes this variant. Reported as pairs of VariationID:classification; multiple values are separated by a vertical bar\">\n",
      "##INFO=<ID=CLNSIGSCV,Number=.,Type=String,Description=\"SCV accession numbers for the submissions that contribute to the aggregate germline classification in ClinVar; multiple values are separated by a vertical bar\">\n",
      "##INFO=<ID=CLNVC,Number=1,Type=String,Description=\"Variant type\">\n",
      "##INFO=<ID=CLNVCSO,Number=1,Type=String,Description=\"Sequence Ontology id for variant type\">\n",
      "##INFO=<ID=CLNVI,Number=.,Type=String,Description=\"the variant's clinical sources reported as tag-value pairs of database and variant identifier\">\n",
      "##INFO=<ID=DBVARID,Number=.,Type=String,Description=\"nsv accessions from dbVar for the variant\">\n",
      "##INFO=<ID=GENEINFO,Number=1,Type=String,Description=\"Gene(s) for the variant reported as gene symbol:gene id. The gene symbol and id are delimited by a colon (:) and each pair is delimited by a vertical bar (|)\">\n",
      "##INFO=<ID=MC,Number=.,Type=String,Description=\"comma separated list of molecular consequence in the form of Sequence Ontology ID|molecular_consequence\">\n",
      "##INFO=<ID=ONCDN,Number=.,Type=String,Description=\"ClinVar's preferred disease name for the concept specified by disease identifiers in ONCDISDB\">\n",
      "##INFO=<ID=ONCDNINCL,Number=.,Type=String,Description=\"For included variant: ClinVar's preferred disease name for the concept specified by disease identifiers in ONCDISDBINCL\">\n",
      "##INFO=<ID=ONCDISDB,Number=.,Type=String,Description=\"Tag-value pairs of disease database name and identifier submitted for oncogenicity classifications, e.g. MedGen:NNNNNN\">\n",
      "##INFO=<ID=ONCDISDBINCL,Number=.,Type=String,Description=\"For included variant: Tag-value pairs of disease database name and identifier for oncogenicity classifications, e.g. OMIM:NNNNNN\">\n",
      "##INFO=<ID=ONC,Number=.,Type=String,Description=\"Aggregate oncogenicity classification for this single variant; multiple values are separated by a vertical bar\">\n",
      "##INFO=<ID=ONCINCL,Number=.,Type=String,Description=\"Oncogenicity classification for a haplotype or genotype that includes this variant. Reported as pairs of VariationID:classification; multiple values are separated by a vertical bar\">\n",
      "##INFO=<ID=ONCREVSTAT,Number=.,Type=String,Description=\"ClinVar review status of oncogenicity classification for the Variation ID\">\n",
      "##INFO=<ID=ONCSCV,Number=.,Type=String,Description=\"SCV accession numbers for the submissions that contribute to the aggregate oncogenicity classification in ClinVar; multiple values are separated by a vertical bar\">\n",
      "##INFO=<ID=ONCCONF,Number=.,Type=String,Description=\"Conflicting oncogenicity classification for this single variant; multiple values are separated by a vertical bar\">\n",
      "##INFO=<ID=ORIGIN,Number=.,Type=String,Description=\"Allele origin. One or more of the following values may be added: 0 - unknown; 1 - germline; 2 - somatic; 4 - inherited; 8 - paternal; 16 - maternal; 32 - de-novo; 64 - biparental; 128 - uniparental; 256 - not-tested; 512 - tested-inconclusive; 1073741824 - other\">\n",
      "##INFO=<ID=RS,Number=.,Type=String,Description=\"dbSNP ID (i.e. rs number)\">\n",
      "##INFO=<ID=SCIDN,Number=.,Type=String,Description=\"ClinVar's preferred disease name for the concept specified by disease identifiers in SCIDISDB\">\n",
      "##INFO=<ID=SCIDNINCL,Number=.,Type=String,Description=\"For included variant: ClinVar's preferred disease name for the concept specified by disease identifiers in SCIDISDBINCL\">\n",
      "##INFO=<ID=SCIDISDB,Number=.,Type=String,Description=\"Tag-value pairs of disease database name and identifier submitted for somatic clinial impact classifications, e.g. MedGen:NNNNNN\">\n",
      "##INFO=<ID=SCIDISDBINCL,Number=.,Type=String,Description=\"For included variant: Tag-value pairs of disease database name and identifier for somatic clinical impact classifications, e.g. OMIM:NNNNNN\">\n",
      "##INFO=<ID=SCIREVSTAT,Number=.,Type=String,Description=\"ClinVar review status of somatic clinical impact for the Variation ID\">\n",
      "##INFO=<ID=SCI,Number=.,Type=String,Description=\"Aggregate somatic clinical impact for this single variant; multiple values are separated by a vertical bar\">\n",
      "##INFO=<ID=SCIINCL,Number=.,Type=String,Description=\"Somatic clinical impact classification for a haplotype or genotype that includes this variant. Reported as pairs of VariationID:classification; multiple values are separated by a vertical bar\">\n",
      "##INFO=<ID=SCISCV,Number=.,Type=String,Description=\"SCV accession numbers for the submissions that contribute to the aggregate somatic clinical impact in ClinVar; multiple values are separated by a vertical bar\">\n",
      "##contig=<ID=1>\n",
      "##contig=<ID=2>\n",
      "##contig=<ID=3>\n",
      "##contig=<ID=4>\n",
      "##contig=<ID=5>\n",
      "##contig=<ID=6>\n",
      "##contig=<ID=7>\n",
      "##contig=<ID=8>\n",
      "##contig=<ID=9>\n",
      "##contig=<ID=10>\n",
      "##contig=<ID=11>\n",
      "##contig=<ID=12>\n",
      "##contig=<ID=13>\n",
      "##contig=<ID=14>\n",
      "##contig=<ID=15>\n",
      "##contig=<ID=16>\n",
      "##contig=<ID=17>\n",
      "##contig=<ID=18>\n",
      "##contig=<ID=19>\n",
      "##contig=<ID=20>\n",
      "##contig=<ID=21>\n",
      "##contig=<ID=22>\n",
      "##contig=<ID=X>\n",
      "##contig=<ID=Y>\n",
      "##contig=<ID=MT>\n",
      "##contig=<ID=NT_113889.1>\n",
      "##contig=<ID=NT_187633.1>\n",
      "##contig=<ID=NT_187661.1>\n",
      "##contig=<ID=NT_187693.1>\n",
      "##contig=<ID=NW_009646201.1>\n",
      "##bcftools_viewVersion=1.19+htslib-1.18\n",
      "##bcftools_viewCommand=view -h /scratch/naimerja/DNASNVData113/clinvar_data/clinvar_grch38.vcf.gz; Date=Fri May  9 12:41:08 2025\n",
      "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\n"
     ]
    }
   ],
   "source": [
    "# Explore ClinVar VCF file structure\n",
    "# Update the file path to point to your ClinVar VCF file\n",
    "\n",
    "import subprocess\n",
    "import os\n",
    "\n",
    "# Example VCF file path (update for your data)\n",
    "vcf_file = \"data/clinvar_grch38.vcf.gz\"  # Update this path\n",
    "\n",
    "# Check if file exists\n",
    "if os.path.exists(vcf_file):\n",
    "    try:\n",
    "        # View VCF header to understand available fields\n",
    "        result = subprocess.run(\n",
    "            [\"bcftools\", \"view\", \"-h\", vcf_file],\n",
    "            capture_output=True, text=True, check=True\n",
    "        )\n",
    "        \n",
    "        print(\"ClinVar VCF Header (first 50 lines):\")\n",
    "        print(\"=\" * 50)\n",
    "        header_lines = result.stdout.split('\\n')[:50]\n",
    "        for line in header_lines:\n",
    "            print(line)\n",
    "            \n",
    "    except (subprocess.CalledProcessError, FileNotFoundError) as e:\n",
    "        print(f\"Error reading VCF file: {e}\")\n",
    "        print(\"Please ensure bcftools is installed and VCF file path is correct\")\n",
    "else:\n",
    "    print(f\"VCF file not found: {vcf_file}\")\n",
    "    print(\"Please update the file path to point to your ClinVar VCF file\")\n",
    "    print(\"Download from: https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/\")\n",
    "\n",
    "print(\"\\nKey ClinVar INFO fields to look for:\")\n",
    "print(\"- CLNSIG: Clinical significance\")\n",
    "print(\"- CLNDN: Disease name\")\n",
    "print(\"- GENEINFO: Gene information\")\n",
    "print(\"- CLNREVSTAT: Review status\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "VEP to clean raw clinvar vcf to cleaned coding only vcf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2) Point to your VEP install and cache, and wire up Perl libs:\n",
    "import os\n",
    "\n",
    "os.environ['VEP_ROOT']   = 'SCRATCH_DIR/DNASNVData113/clinvar_data/vep-code-113'\n",
    "os.environ['VEP_CACHE']  = 'SCRATCH_DIR/DNASNVData113/clinvar_data/vep-cache-113'\n",
    "os.environ['PERL5LIB']   = 'SCRATCH_DIR/perl5/lib/perl5:' + os.environ.get('PERL5LIB','')\n",
    "# prepend VEP_ROOT onto the existing PATH\n",
    "os.environ['PATH']       = os.environ['VEP_ROOT'] + ':' + os.environ.get('PATH','')\n",
    "\n",
    "# now this will actually show your full, correct PATH:\n",
    "!echo $PATH\n",
    "!which bash\n",
    "!which vep\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "/usr/bin/time -v $VEP_ROOT/vep \\\n",
    "  --input_file  SCRATCH_DIR/DNASNVData113/clinvar_data/clinvar_grch38.vcf.gz \\\n",
    "  --output_file SCRATCH_DIR/DNASNVData113/clinvar_data/clinvar_coding_only.vcf \\\n",
    "  --cache \\\n",
    "  --dir_cache $VEP_CACHE \\\n",
    "  --offline \\\n",
    "  --fasta $VEP_CACHE/homo_sapiens/113_GRCh38/Homo_sapiens.GRCh38.dna.toplevel.fa \\\n",
    "  --species homo_sapiens \\\n",
    "  --assembly GRCh38 \\\n",
    "  --vcf \\\n",
    "  --hgvs \\\n",
    "  --pick \\\n",
    "  --fork 48 \\\n",
    "  --force_overwrite \\\n",
    "  --verbose \\\n",
    "  --coding_only\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: VEP Processing\n",
    "\n",
    "Process ClinVar VCF through VEP to add annotations and filter for coding variants."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!/usr/bin/env python3\n",
    "import hgvs.edit as HEdit\n",
    "from hgvs.parser import Parser\n",
    "from hgvs.exceptions import HGVSError\n",
    "from hgvs.enums import Datum\n",
    "import hgvs.location as loc\n",
    "\n",
    "from collections import Counter\n",
    "from concurrent.futures import ProcessPoolExecutor\n",
    "from tqdm import tqdm\n",
    "\n",
    "def is_coding_pos(pos):\n",
    "    \"\"\"\n",
    "    Return True if the given position is within the translated CDS.\n",
    "    Excludes:\n",
    "      - intronic offsets (BaseOffsetPosition.is_intronic)\n",
    "      - 5′ UTR (datum=CDS_START and base < 1)\n",
    "      - 3′ UTR    (datum=CDS_END)\n",
    "    \"\"\"\n",
    "    p = pos.start if hasattr(pos, \"start\") else pos\n",
    "    if isinstance(p, loc.BaseOffsetPosition):\n",
    "        dbg = f\"(base={p.base}, datum={p.datum}, offset={p.offset})\"\n",
    "        if p.is_intronic:\n",
    "            return False\n",
    "        if p.datum == Datum.CDS_START and p.base < 1:\n",
    "            return False\n",
    "        if p.datum == Datum.CDS_END:\n",
    "            return False\n",
    "        if p.datum == Datum.CDS_START and p.base >= 1:\n",
    "            return True\n",
    "        # any other datum we don’t recognize\n",
    "        raise ValueError(f\"Unrecognized BaseOffsetPosition {dbg}, full pos object: {pos!r}\")\n",
    "\n",
    "def _init_worker(idx):\n",
    "    # runs once in each worker\n",
    "    global parser, hgvsc_idx\n",
    "    parser    = Parser()\n",
    "    hgvsc_idx = idx\n",
    "\n",
    "\n",
    "def _classify_line(line):\n",
    "    # split on tabs to get INFO (column 7)\n",
    "    cols = line.rstrip(\"\\n\").split(\"\\t\")\n",
    "    if len(cols) < 8:\n",
    "        return (\"unmatched\", None, \"\")\n",
    "\n",
    "    info = cols[7]\n",
    "    # pull CSQ=\n",
    "    csq_entries = [kv.split(\"=\",1)[1]\n",
    "                   for kv in info.split(\";\")\n",
    "                   if kv.startswith(\"CSQ=\")]\n",
    "    if not csq_entries:\n",
    "        return (\"unmatched\", None, \"\")\n",
    "\n",
    "    # first allele in CSQ, then HGVSc field\n",
    "    hfull = csq_entries[0].split(\",\")[0].split(\"|\")[hgvsc_idx]\n",
    "    if not hfull:\n",
    "        return (\"unmatched\", None, \"\")\n",
    "\n",
    "    # parse HGVS\n",
    "    try:\n",
    "        var = parser.parse_hgvs_variant(hfull)\n",
    "    except HGVSError:\n",
    "        return (\"unmatched\", None, hfull)\n",
    "\n",
    "    edit = var.posedit.edit\n",
    "    pos  = var.posedit.pos\n",
    "\n",
    "    # get 1-based start/end\n",
    "    if hasattr(pos, \"start\") and hasattr(pos, \"end\"):\n",
    "        start = pos.start.base\n",
    "        end   = pos.end.base\n",
    "    else:\n",
    "        start = end = pos.base\n",
    "\n",
    "    # generic type key\n",
    "    etype = edit.type  # attribute, not method\n",
    "    if etype in (\"del\", \"dup\", \"inv\"):\n",
    "        key = f\"{etype}_{'single' if start == end else 'range'}\"\n",
    "    else:\n",
    "        key = etype    # covers sub, ins, delins, etc.\n",
    "\n",
    "    # coding vs noncoding\n",
    "    coding = is_coding_pos(pos)\n",
    "\n",
    "    return (key, coding, None)\n",
    "\n",
    "\n",
    "def scan_hgvsc_types(vcf_path, max_workers=24):\n",
    "    # 1) find CSQ header → HGVSc index\n",
    "    csq_fields = None\n",
    "    with open(vcf_path) as f:\n",
    "        for line in f:\n",
    "            if line.startswith(\"##INFO=<ID=CSQ\"):\n",
    "                desc = line.split(\"Format:\")[1].split('\">')[0].strip()\n",
    "                csq_fields = desc.split(\"|\")\n",
    "                break\n",
    "    if not csq_fields:\n",
    "        raise RuntimeError(\"Couldn't find CSQ header in VCF\")\n",
    "    idx = csq_fields.index(\"HGVSc\")\n",
    "\n",
    "    # 2) count lines for progress bar\n",
    "    total = sum(1 for _ in open(vcf_path) if not _.startswith(\"#\"))\n",
    "\n",
    "    coding_counts    = Counter()\n",
    "    noncoding_counts = Counter()\n",
    "    unmatched_counts = Counter()\n",
    "\n",
    "    # 3) parallel processing\n",
    "    with ProcessPoolExecutor(\n",
    "        max_workers=max_workers,\n",
    "        initializer=_init_worker,\n",
    "        initargs=(idx,)\n",
    "    ) as exe:\n",
    "        # only non-header lines\n",
    "        lines = (l for l in open(vcf_path) if not l.startswith(\"#\"))\n",
    "        for key, coding, extra in tqdm(\n",
    "            exe.map(_classify_line, lines, chunksize=1000),\n",
    "            total=total,\n",
    "            desc=\"Scanning variants\"\n",
    "        ):\n",
    "            if key == \"unmatched\":\n",
    "                unmatched_counts[extra] += 1\n",
    "            else:\n",
    "                if coding:\n",
    "                    coding_counts[key] += 1\n",
    "                else:\n",
    "                    noncoding_counts[key] += 1\n",
    "\n",
    "    # 4) report\n",
    "    print(\"\\n=== Coding-region variants ===\")\n",
    "    for name, cnt in coding_counts.most_common():\n",
    "        print(f\"  {name}: {cnt}\")\n",
    "\n",
    "    print(\"\\n=== Non-coding variants (UTR & intronic) ===\")\n",
    "    for name, cnt in noncoding_counts.most_common():\n",
    "        print(f\"  {name}: {cnt}\")\n",
    "\n",
    "    print(\"\\n=== Unmatched HGVSc patterns ===\")\n",
    "    for h, cnt in unmatched_counts.most_common():\n",
    "        print(f\"  {h}: {cnt}\")\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    scan_hgvsc_types(\n",
    "        \"SCRATCH_DIR/DNASNVData113/clinvar_data/clinvar_coding_only.vcf\",\n",
    "        max_workers=24\n",
    "    )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating data table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!/usr/bin/env python3\n",
    "import os\n",
    "import pandas as pd\n",
    "# Use 24 threads for PyArrow encoding\n",
    "os.environ[\"ARROW_NUM_THREADS\"] = \"24\"\n",
    "\n",
    "import pysam\n",
    "import pyarrow as pa\n",
    "import pyarrow.parquet as pq\n",
    "from tqdm import tqdm\n",
    "\n",
    "def get_window(genome, chrom, pos0, window_size=4096, pad_char=\"N\"):\n",
    "    \"\"\"\n",
    "    Fetch exactly `window_size` bases centered at 0-based pos0\n",
    "    from the pysam.FastaFile `genome`, padding with `pad_char`.\n",
    "    \"\"\"\n",
    "    half  = window_size // 2\n",
    "    start = pos0 - half\n",
    "    end   = start + window_size\n",
    "\n",
    "    parts = []\n",
    "    chrom_len = genome.get_reference_length(chrom)\n",
    "\n",
    "    # left padding\n",
    "    if start < 0:\n",
    "        parts.append(pad_char * -start)\n",
    "        fetch_start = 0\n",
    "    else:\n",
    "        fetch_start = start\n",
    "\n",
    "    # fetch middle\n",
    "    fetch_end = min(end, chrom_len)\n",
    "    parts.append(genome.fetch(chrom, fetch_start, fetch_end))\n",
    "\n",
    "    # right padding\n",
    "    if fetch_end < end:\n",
    "        parts.append(pad_char * (end - fetch_end))\n",
    "\n",
    "    return \"\".join(parts)\n",
    "\n",
    "\n",
    "def main(vcf_path, genome_fasta_path, out_parquet_path):\n",
    "    use_cols = [\"symbol\", \"name\", \"entrez_id\"]\n",
    "    hgnc_df = pd.read_csv(\n",
    "        \"SCRATCH_DIR/DNASNVData113/clinvar_data/hgnc_complete_set.txt\",\n",
    "        sep=\"\\t\", usecols=use_cols,\n",
    "        dtype={\"entrez_id\": \"Int64\"}\n",
    "    )\n",
    "    # build a dict mapping Entrez ID → approved name\n",
    "    gene_desc_map = dict(zip(\n",
    "        hgnc_df[\"entrez_id\"].astype(str),  # ensure keys are strings if your gene_id is str\n",
    "        hgnc_df[\"name\"]\n",
    "    ))\n",
    "\n",
    "    missing_genes = 0\n",
    "    # definitions\n",
    "    PATHOGENIC_ALLOWED = {\n",
    "        \"pathogenic\",\n",
    "        \"pathogenic/likely_pathogenic\",\n",
    "        \"likely_pathogenic\",\n",
    "        \"benign\",\n",
    "        \"likely_benign\",\n",
    "        \"benign/likely_benign\",\n",
    "    }\n",
    "\n",
    "    REVIEW_STATUS_ALLOWED = {\n",
    "        \"criteria_provided,_multiple_submitters,_no_conflicts\",\n",
    "        \"reviewed_by_expert_panel\",\n",
    "        \"practice_guideline\",\n",
    "    }\n",
    "\n",
    "    # 0) explicitly remove any old output\n",
    "    try:\n",
    "        os.remove(out_parquet_path)\n",
    "    except FileNotFoundError:\n",
    "        pass\n",
    "\n",
    "    # count variants for progress bar\n",
    "    total = sum(1 for line in open(vcf_path) if not line.startswith(\"#\"))\n",
    "\n",
    "    # open the genomic FASTA\n",
    "    genome = pysam.FastaFile(genome_fasta_path)\n",
    "    fasta_contigs = set(genome.references)  # <<< build this once\n",
    "\n",
    "    # prepare for Parquet writing\n",
    "    writer = None\n",
    "    batch = {col: [] for col in (\n",
    "        \"clinvar_id\",\n",
    "        \"original_window\",\n",
    "        \"mutated_window\",\n",
    "        \"cleaned_pathogenicity\",\n",
    "        \"disease_name\",\n",
    "        \"gene_name\",\n",
    "        \"gene_desc\",\n",
    "        \"chromosome\",\n",
    "        \"chromosome_position\",\n",
    "        \"variant_type\",\n",
    "        \"clinvar_link\",\n",
    "        \"gene_id\",\n",
    "        \"mutation_instruction\",\n",
    "        \"pathogenicity\",\n",
    "        \"review_status\"\n",
    "    )}\n",
    "    batch_size = 100_000\n",
    "\n",
    "    def flush_batch():\n",
    "        nonlocal writer, batch\n",
    "        table = pa.Table.from_pydict(batch)\n",
    "        if writer is None:\n",
    "            writer = pq.ParquetWriter(\n",
    "                out_parquet_path,\n",
    "                table.schema,\n",
    "                compression=\"snappy\",\n",
    "                use_dictionary=True\n",
    "            )\n",
    "        writer.write_table(table)\n",
    "        for col in batch:\n",
    "            batch[col].clear()\n",
    "\n",
    "    # process VCF\n",
    "    with open(vcf_path) as vf:\n",
    "        for line in tqdm(vf, total=total, desc=\"Writing Parquet\"):\n",
    "            if line.startswith(\"#\"):\n",
    "                continue\n",
    "            cols = line.rstrip(\"\\n\").split(\"\\t\")\n",
    "            chrom, pos1, clinvar_id, ref, alt = cols[:5]\n",
    "\n",
    "            # --- SKIP if this contig is not in your FASTA --- or mitochondrial chromosome (keeps only nuclear chromosomes as in Evo2)\n",
    "            if chrom not in fasta_contigs or chrom == \"MT\":\n",
    "                continue\n",
    "\n",
    "            # Skip variants too large to fit sensibly in a 4 096 bp window\n",
    "            MAX_EDIT = 64 # 64 bp\n",
    "            if len(ref) > MAX_EDIT or len(alt) > MAX_EDIT:\n",
    "                continue\n",
    "\n",
    "\n",
    "            info = {\n",
    "                kv.split(\"=\", 1)[0]: kv.split(\"=\", 1)[1]\n",
    "                for kv in cols[7].split(\";\") if \"=\" in kv\n",
    "            }\n",
    "\n",
    "            # mutation instruction\n",
    "            instr = f\"{ref}>{alt}\"\n",
    "\n",
    "            # extract 4096-bp window\n",
    "            pos0 = int(pos1) - 1\n",
    "            orig_win = get_window(genome, chrom, pos0, window_size=4096)\n",
    "\n",
    "            # apply REF→ALT at center\n",
    "            half = 4096 // 2\n",
    "            i0   = half\n",
    "            i1   = half + len(ref)\n",
    "            mut_win = orig_win[:i0] + alt + orig_win[i1:]\n",
    "            # enforce fixed length\n",
    "            if len(mut_win) < 4096:\n",
    "                mut_win = mut_win.ljust(4096, \"N\")\n",
    "            elif len(mut_win) > 4096:\n",
    "                mut_win = mut_win[:4096]\n",
    "\n",
    "            # pathogenicity, disease, variant type\n",
    "            path = info.get(\"CLNSIG\", \"\").lower()\n",
    "            dis  = info.get(\"CLNDN\", \"\")\n",
    "            gene_info = info.get(\"GENEINFO\", \"\")\n",
    "\n",
    "            #filter out variants with no gene info\n",
    "            if gene_info ==\"\":\n",
    "                missing_genes +=1\n",
    "                continue\n",
    "            else:\n",
    "                gene_name = gene_info.split(\":\")[0]\n",
    "                gene_id = gene_info.split(\":\")[1]\n",
    "\n",
    "\n",
    "            vart = \"SNV\" if len(ref) == 1 == len(alt) else \"non_SNV\"\n",
    "            rev_stat = info.get(\"CLNREVSTAT\", \"\").lower()\n",
    "\n",
    "            # filter for pathogenic/(|)likely pathogenic or benign/(|)likely benign only\n",
    "            # only keep if ANY of the pipe-delimited terms is in our allowed set\n",
    "            terms = path.split(\"|\")\n",
    "            if not any(term in PATHOGENIC_ALLOWED for term in terms):\n",
    "                continue\n",
    "\n",
    "            # filter for review status\n",
    "            if rev_stat not in REVIEW_STATUS_ALLOWED:\n",
    "                continue\n",
    "\n",
    "            if \"pathogenic\" in path:\n",
    "                clean_pathogenicity = \"pathogenic\"\n",
    "            elif \"benign\" in path:\n",
    "                clean_pathogenicity = \"benign\"\n",
    "            else:\n",
    "                raise ValueError(f\"Unknown pathogenicity: {path}\")\n",
    "\n",
    "\n",
    "            # collect row\n",
    "            batch[\"clinvar_id\"].append(clinvar_id)\n",
    "            batch[\"mutation_instruction\"].append(instr)\n",
    "            batch[\"original_window\"].append(orig_win)\n",
    "            batch[\"mutated_window\"].append(mut_win)\n",
    "            batch[\"pathogenicity\"].append(path)\n",
    "            batch[\"cleaned_pathogenicity\"].append(clean_pathogenicity)\n",
    "            batch[\"disease_name\"].append(dis)\n",
    "            batch[\"variant_type\"].append(vart)\n",
    "            batch[\"review_status\"].append(rev_stat)\n",
    "            batch[\"gene_name\"].append(gene_name)\n",
    "            batch[\"gene_id\"].append(gene_id)\n",
    "            batch[\"chromosome\"].append(chrom)\n",
    "            batch[\"chromosome_position\"].append(pos1) # 1-based position on chromosome\n",
    "            batch[\"gene_desc\"].append(gene_desc_map.get(gene_id))\n",
    "            batch[\"clinvar_link\"].append(f\"https://www.ncbi.nlm.nih.gov/clinvar/variation/{clinvar_id}/\")\n",
    "\n",
    "            # flush when batch is full\n",
    "            if len(batch[\"mutation_instruction\"]) >= batch_size:\n",
    "                flush_batch()\n",
    "\n",
    "    # final flush & close\n",
    "    if batch[\"mutation_instruction\"]:\n",
    "        flush_batch()\n",
    "    if writer is not None:\n",
    "        writer.close()\n",
    "\n",
    "    print(\"Finished writing →\", out_parquet_path)\n",
    "    print(f\"# Removed due to missing gene info: {missing_genes}\")\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main(\n",
    "        \"SCRATCH_DIR/DNASNVData113/clinvar_data/clinvar_coding_only.vcf\",\n",
    "        \"SCRATCH_DIR/DNASNVData113/clinvar_data/\"\n",
    "        \"vep-cache-113/homo_sapiens/113_GRCh38/\"\n",
    "        \"Homo_sapiens.GRCh38.dna.toplevel.fa\",\n",
    "        \"SCRATCH_DIR/DNASNVData113/clinvar_data/\"\n",
    "        \"clinvar_windowed_4096.parquet\"\n",
    "    )\n",
    "\n",
    "# note to visually inspect the dna sequences and modified sequences go to https://www.ncbi.nlm.nih.gov/gdv/browser/genome/?id=GCF_000001405.40 and then click tools and then sequence text view"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hereditary_factor_VIII_deficiency_disease|not_provided'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df['clinvar_id']=='10152']['disease_name'][342667]\n",
    "# https://www.ncbi.nlm.nih.gov/clinvar/variation/10152/\n",
    "# shows that only diseases with stars are included in the associated diseases (since hemophelia not included)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[print(x) for x in (df[(df['pathogenicity']=='pathogenic') & df['disease_name'].str.contains(r'\\|')]['clinvar_link'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On login node upload table to huggingface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --no-index huggingface-hub\n",
    "from huggingface_hub import HfApi\n",
    "import os\n",
    "import glob\n",
    "\n",
    "# 0) config\n",
    "repo_id     = \"wanglab/bioR_tasks\"         # your dataset repo\n",
    "repo_type   = \"dataset\"\n",
    "subfolder   = \"variant_effect_non_snv_and_snv\"\n",
    "local_dir   = \"SCRATCH_DIR/DNASNVData113/clinvar_data/clinvar_windowed_4096.parquet\"\n",
    "\n",
    "api = HfApi()\n",
    "\n",
    "# 1) list all files in that subfolder\n",
    "all_files = api.list_repo_files(repo_id, repo_type=repo_type)\n",
    "old_files = [f for f in all_files if f.startswith(subfolder + \"/\")]\n",
    "\n",
    "print(f\"Will delete {len(old_files)} old files:\")\n",
    "for f in old_files:\n",
    "    print(\"  \", f)\n",
    "\n",
    "# 2) delete them (one commit per file, or you can batch by reusing the same commit_message)\n",
    "for f in old_files:\n",
    "    api.delete_file(\n",
    "        path_in_repo = f,\n",
    "        repo_id      = repo_id,\n",
    "        repo_type    = repo_type,\n",
    "        commit_message = f\"remove old dataset file\"\n",
    "    )\n",
    "\n",
    "# 3) upload your single Parquet file\n",
    "new_file = \"SCRATCH_DIR/DNASNVData113/clinvar_data/clinvar_windowed_4096.parquet\"\n",
    "basename = os.path.basename(new_file)\n",
    "dest_path = f\"{subfolder}/{basename}\"\n",
    "\n",
    "print(f\"Uploading {new_file!r} to {repo_id}/{dest_path} …\")\n",
    "api.upload_file(\n",
    "    path_or_fileobj = new_file,\n",
    "    path_in_repo    = dest_path,\n",
    "    repo_id         = repo_id,\n",
    "    repo_type       = repo_type,\n",
    "    commit_message  = f\"add updated parquet {basename}\"\n",
    ")\n",
    "\n",
    "print(\"Done! Your dataset has been updated on the Hub.\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --no-index huggingface-hub\n",
    "from huggingface_hub import HfApi\n",
    "import os\n",
    "import glob\n",
    "\n",
    "# 0) config\n",
    "repo_id     = \"wanglab/bioR_tasks\"         # your dataset repo\n",
    "repo_type   = \"dataset\"\n",
    "subfolder   = \"variant_effect_non_snv_and_snv\"\n",
    "local_dir   = \"SCRATCH_DIR/DNASNVData113/clinvar_data/clinvar_windowed_4096.parquet\"\n",
    "\n",
    "api = HfApi()\n",
    "\n",
    "# 1) list all files in that subfolder\n",
    "all_files = api.list_repo_files(repo_id, repo_type=repo_type)\n",
    "old_files = [f for f in all_files if f.startswith(subfolder + \"/\")]\n",
    "\n",
    "\n",
    "import io\n",
    "\n",
    "# Upload cleaned DataFrame\n",
    "buffer = io.BytesIO()\n",
    "final_df.to_parquet(buffer, index=False)\n",
    "buffer.seek(0)\n",
    "\n",
    "# Construct cleaned filename by appending '_cleaned'\n",
    "basename = os.path.splitext(os.path.basename(local_dir))[0] + \"_cleaned.parquet\"\n",
    "dest_path = f\"{subfolder}/{basename}\"\n",
    "\n",
    "print(f\"Uploading cleaned DataFrame to {repo_id}/{dest_path} …\")\n",
    "api.upload_file(\n",
    "    path_or_fileobj=buffer,\n",
    "    path_in_repo=dest_path,\n",
    "    repo_id=repo_id,\n",
    "    repo_type=repo_type,\n",
    "    commit_message=f\"add cleaned parquet {basename}\"\n",
    ")\n",
    "\n",
    "print(\"Done! Cleaned DataFrame uploaded.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "read table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "→ Discovering data under '/scratch/naimerja/DNASNVData113/clinvar_data/clinvar_windowed_4096.parquet'\n",
      "→ Scanning & reading all fragments in parallel …\n",
      "→ Converting to pandas DataFrame…\n",
      "✅ Loaded 342,689 rows in 3.3s\n",
      "DataFrame shape: (342689, 15)\n",
      "Memory usage: 3.18 GB\n"
     ]
    }
   ],
   "source": [
    "#!/usr/bin/env python3\n",
    "import time, os\n",
    "import pandas as pd\n",
    "import pyarrow as pa\n",
    "import pyarrow.parquet as pq\n",
    "import pyarrow.dataset as ds\n",
    "\n",
    "from tqdm import tqdm\n",
    "\n",
    "def load_parquet_to_pandas(parquet_dir, num_threads=24):\n",
    "    # configure PyArrow global thread pool\n",
    "    pa.set_cpu_count(num_threads)\n",
    "    pa.set_io_thread_count(num_threads)\n",
    "\n",
    "    start = time.time()\n",
    "    print(f\"→ Discovering data under {parquet_dir!r}\")\n",
    "\n",
    "    # Option A: use the ParquetDataset API\n",
    "    # dataset = pq.ParquetDataset(parquet_dir)      # older PyArrow\n",
    "    # table   = dataset.read(use_threads=True)      # uses all threads by default\n",
    "\n",
    "    # Option B (recommended): use the Dataset API\n",
    "    dataset = ds.dataset(parquet_dir, format=\"parquet\")\n",
    "    print(\"→ Scanning & reading all fragments in parallel …\")\n",
    "    # to_table will read all row-groups/files in parallel (use_threads defaults to True) :contentReference[oaicite:0]{index=0}\n",
    "    table = dataset.to_table()\n",
    "\n",
    "    print(\"→ Converting to pandas DataFrame…\")\n",
    "    df = table.to_pandas()\n",
    "\n",
    "    end = time.time()\n",
    "    print(f\"✅ Loaded {len(df):,} rows in {end - start:.1f}s\")\n",
    "    print(f\"DataFrame shape: {df.shape}\")\n",
    "    print(f\"Memory usage: {df.memory_usage(deep=True).sum() / 1e9:.2f} GB\")\n",
    "\n",
    "    return df\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    PARQUET_DIR = \"SCRATCH_DIR/DNASNVData113/clinvar_data/clinvar_windowed_4096.parquet\"\n",
    "    df = load_parquet_to_pandas(PARQUET_DIR, num_threads=24)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create final training dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import re\n",
    "\n",
    "#list of 50 questions\n",
    "\n",
    "question_synonyms = {\n",
    "    \"A genetic variant on chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, affects the gene <GENE_SYMBOL> (<GENE_FULL_NAME>). Is this variant benign or pathogenic? If pathogenic, what disease(s) does it cause?\",\n",
    "    \"A mutation at chromosome position <CHROMOSOME_POSITION> on chromosome <CHROMOSOME_NUMBER> in gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? If pathogenic, which disease(s) is it linked to?\",\n",
    "    \"Considering the variant on chromosome <CHROMOSOME_NUMBER>, location <CHROMOSOME_POSITION>, involving gene <GENE_SYMBOL> (<GENE_FULL_NAME>), would you classify it as benign or pathogenic? What disease(s), if any, does a pathogenic variant indicate?\",\n",
    "    \"Is the genetic mutation found on chromosome <CHROMOSOME_NUMBER> at position <CHROMOSOME_POSITION>, within the gene <GENE_SYMBOL> (<GENE_FULL_NAME>), considered benign or pathogenic? If pathogenic, specify the associated disease(s).\",\n",
    "    \"Assess the clinical significance (benign or pathogenic) of the variant at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>). What disease(s) is it linked to if pathogenic?\",\n",
    "    \"Does the genetic variant at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, impacting gene <GENE_SYMBOL> (<GENE_FULL_NAME>), appear benign or pathogenic? If pathogenic, name the associated disease(s).\",\n",
    "    \"Variant in gene <GENE_SYMBOL> (<GENE_FULL_NAME>), located at chromosome <CHROMOSOME_NUMBER> position <CHROMOSOME_POSITION>: benign or pathogenic? What disease(s) does it cause if pathogenic?\",\n",
    "    \"Gene <GENE_SYMBOL> (<GENE_FULL_NAME>) variant at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>—is it benign or pathogenic? If pathogenic, what are the associated condition(s)?\",\n",
    "    \"A genetic alteration at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, in gene <GENE_SYMBOL> (<GENE_FULL_NAME>)—benign or pathogenic? If pathogenic, which disease(s) is involved?\",\n",
    "    \"Chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>): Is this mutation clinically benign or pathogenic? If pathogenic, identify the related disease(s).\",\n",
    "    \"Does the variant on chromosome <CHROMOSOME_NUMBER> at location <CHROMOSOME_POSITION> affecting gene <GENE_SYMBOL> (<GENE_FULL_NAME>) have a clinical significance of benign or pathogenic? If pathogenic, what disease(s) is associated?\",\n",
    "    \"Mutation at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, within <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? If pathogenic, indicate the disease(s).\",\n",
    "    \"Evaluate this variant at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? If pathogenic, what are the disease connection(s)?\",\n",
    "    \"Gene mutation in <GENE_SYMBOL> (<GENE_FULL_NAME>) at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>—is it benign or pathogenic? If pathogenic, specify the disease(s).\",\n",
    "    \"Located at chromosome <CHROMOSOME_NUMBER> position <CHROMOSOME_POSITION>, the variant affecting gene <GENE_SYMBOL> (<GENE_FULL_NAME>)—benign or pathogenic? If pathogenic, which disease(s) does it relate to?\",\n",
    "    \"Is the chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION> variant in <GENE_SYMBOL> (<GENE_FULL_NAME>) clinically benign or pathogenic? If pathogenic, what condition(s) is associated?\",\n",
    "    \"Clinical significance of chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? Name the disease(s) if pathogenic.\",\n",
    "    \"Is the genetic variant on chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>), benign or pathogenic? If pathogenic, what disease(s) is indicated?\",\n",
    "    \"Regarding the variant at chromosome <CHROMOSOME_NUMBER> and position <CHROMOSOME_POSITION>, affecting gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? If pathogenic, what are the associated illness(es)?\",\n",
    "    \"The mutation in gene <GENE_SYMBOL> (<GENE_FULL_NAME>) at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>—clinically benign or pathogenic? If pathogenic, identify the related disease(s).\",\n",
    "    \"Assess the variant on chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, impacting <GENE_SYMBOL> (<GENE_FULL_NAME>): is it benign or pathogenic? If pathogenic, specify the associated condition(s).\",\n",
    "    \"Variant in <GENE_SYMBOL> (<GENE_FULL_NAME>), chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>—is this benign or pathogenic? If pathogenic, what disease(s) is linked?\",\n",
    "    \"Clinical impact (benign or pathogenic) of the variant at chromosome <CHROMOSOME_NUMBER>, location <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>): what disease(s) if pathogenic?\",\n",
    "    \"The chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION> genetic variant in gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? If pathogenic, indicate disease(s).\",\n",
    "    \"Determine if the mutation at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION> in gene <GENE_SYMBOL> (<GENE_FULL_NAME>) is benign or pathogenic. If pathogenic, what disease(s) is associated?\",\n",
    "    \"Is chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>) variant benign or pathogenic? If pathogenic, what condition(s) is it related to?\",\n",
    "    \"The mutation impacting <GENE_SYMBOL> (<GENE_FULL_NAME>) on chromosome <CHROMOSOME_NUMBER> at position <CHROMOSOME_POSITION>: benign or pathogenic? Name the associated disease(s) if pathogenic.\",\n",
    "    \"Variant at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>): clinically benign or pathogenic? If pathogenic, specify the disease(s) involved.\",\n",
    "    \"Chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic variant? If pathogenic, what are the linked illness(es)?\",\n",
    "    \"A genetic variant at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, affecting gene <GENE_SYMBOL> (<GENE_FULL_NAME>)—is it benign or pathogenic? If pathogenic, identify the associated disorder(s).\",\n",
    "    \"Mutation found at chromosome <CHROMOSOME_NUMBER> position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? If pathogenic, indicate the relevant disease(s).\",\n",
    "    \"Benign or pathogenic: chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>) variant? Disease(s) if pathogenic?\",\n",
    "    \"Evaluate if the mutation on chromosome <CHROMOSOME_NUMBER> at position <CHROMOSOME_POSITION> in <GENE_SYMBOL> (<GENE_FULL_NAME>) is benign or pathogenic. Disease name(s) if pathogenic?\",\n",
    "    \"Clinical classification of chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? Disease(s) if pathogenic?\",\n",
    "    \"Variant chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? Disease(s)?\",\n",
    "    \"Variant on chromosome <CHROMOSOME_NUMBER>, at position <CHROMOSOME_POSITION>, affecting <GENE_SYMBOL> (<GENE_FULL_NAME>): is it benign or pathogenic? If pathogenic, specify the associated disease(s).\",\n",
    "    \"Does the chromosome <CHROMOSOME_NUMBER> mutation at position <CHROMOSOME_POSITION> within gene <GENE_SYMBOL> (<GENE_FULL_NAME>) classify as benign or pathogenic? If pathogenic, indicate the related illness(es).\",\n",
    "    \"Determine whether the variant at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, in gene <GENE_SYMBOL> (<GENE_FULL_NAME>) is benign or pathogenic. If pathogenic, identify the relevant disease(s).\",\n",
    "    \"Gene <GENE_SYMBOL> (<GENE_FULL_NAME>) variant at chromosome position <CHROMOSOME_POSITION> on chromosome <CHROMOSOME_NUMBER>: benign or pathogenic? If pathogenic, what disease(s) is it associated with?\",\n",
    "    \"Considering the genetic mutation at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, impacting <GENE_SYMBOL> (<GENE_FULL_NAME>): is it clinically benign or pathogenic? Name the associated disease(s) if pathogenic.\",\n",
    "    \"Evaluate the clinical significance of the mutation at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION> in gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? What disease(s) does a pathogenic variant suggest?\",\n",
    "    \"Is the variant located on chromosome <CHROMOSOME_NUMBER> at position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>), benign or pathogenic? If pathogenic, specify the disease(s) linked.\",\n",
    "    \"Classify the chromosome <CHROMOSOME_NUMBER> variant at position <CHROMOSOME_POSITION> affecting gene <GENE_SYMBOL> (<GENE_FULL_NAME>) as benign or pathogenic. If pathogenic, which disease(s) is associated?\",\n",
    "    \"For chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic mutation? If pathogenic, what are the associated disease(s)?\",\n",
    "    \"Is the genetic change at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, within gene <GENE_SYMBOL> (<GENE_FULL_NAME>) benign or pathogenic? Name the disease(s) if pathogenic.\",\n",
    "    \"Does the variant impacting <GENE_SYMBOL> (<GENE_FULL_NAME>) on chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, classify as benign or pathogenic? If pathogenic, what disease(s) is it associated with?\",\n",
    "    \"Variant at chromosome position <CHROMOSOME_POSITION>, chromosome <CHROMOSOME_NUMBER>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? If pathogenic, what condition(s) does it relate to?\",\n",
    "    \"Regarding the variant found on chromosome <CHROMOSOME_NUMBER> at position <CHROMOSOME_POSITION> in gene <GENE_SYMBOL> (<GENE_FULL_NAME>): is it benign or pathogenic? If pathogenic, identify the disease(s).\",\n",
    "    \"The genetic variant at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, affecting gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? Disease name(s) if pathogenic?\",\n",
    "    \"Clinically, how would you classify the variant at chromosome <CHROMOSOME_NUMBER>, position <CHROMOSOME_POSITION>, gene <GENE_SYMBOL> (<GENE_FULL_NAME>): benign or pathogenic? If pathogenic, specify the associated illness(es).\"\n",
    "}\n",
    "\n",
    "question_df = pd.DataFrame({'question': list(question_synonyms)})\n",
    "question_df.index.name = 'question_number'\n",
    "\n",
    "# copy the df to training_df\n",
    "training_df = df.copy()\n",
    "training_df = training_df.rename(columns={'original_window': 'reference_sequence', 'mutated_window': 'mutated_sequence'})\n",
    "training_df['question_number'] = np.random.randint(0, 50, size=len(training_df)) # generate random question number between 0 and 49 inclusive\n",
    "\n",
    "# merge the training_df with the question_df\n",
    "training_df = pd.merge(training_df, question_df, on='question_number', how='left')\n",
    "\n",
    "# drop the question_number column\n",
    "training_df = training_df.drop(columns=['question_number'])\n",
    "\n",
    "def fill_placeholders(row):\n",
    "    q = row['question']\n",
    "    # always replace these\n",
    "    q = q.replace('<CHROMOSOME_NUMBER>', str(row['chromosome']))\n",
    "    q = q.replace('<CHROMOSOME_POSITION>', str(row['chromosome_position']))\n",
    "    q = q.replace('<GENE_SYMBOL>', row['gene_name'])\n",
    "    \n",
    "    # gene_full_name may be None\n",
    "    if pd.notnull(row['gene_desc']):\n",
    "        q = q.replace('<GENE_FULL_NAME>', row['gene_desc'])\n",
    "    else:\n",
    "        # remove the entire \"(<GENE_FULL_NAME>)\" including surrounding space\n",
    "        q = re.sub(r'\\s*\\(\\s*<GENE_FULL_NAME>\\s*\\)', '', q)\n",
    "    \n",
    "    return q\n",
    "\n",
    "training_df['question'] = training_df.apply(fill_placeholders, axis=1)\n",
    "\n",
    "\n",
    "\n",
    "def format_answer(row):\n",
    "    path = row['cleaned_pathogenicity']\n",
    "    disease = row['disease_name']\n",
    "    \n",
    "    # If disease_name is exactly 'not_provided' or 'not_specified'\n",
    "    if disease in ('not_provided', 'not_specified', 'not_specified|not_provided', 'not_provided|not_specified'):\n",
    "        return path\n",
    "    \n",
    "    # Split on '|' into a list and drop 'not_provided'\n",
    "    diseases = [d for d in disease.split('|') if d != 'not_provided']\n",
    "    \n",
    "    # Handle 'not_specified': note it, then drop it\n",
    "    unspecified = 'not_specified' in diseases\n",
    "    diseases = [d for d in diseases if d != 'not_specified']\n",
    "    \n",
    "    # Sort the disease names alphabetically\n",
    "    diseases = sorted(diseases)\n",
    "    \n",
    "    # If unspecified, append the note as an element at the end\n",
    "    if unspecified:\n",
    "        diseases.append('likely other unspecified diseases')\n",
    "    \n",
    "    # Represent diseases as a Python-style list literal\n",
    "    disease_text = str(diseases)  # e.g. \"['DiseaseA', 'DiseaseB']\"\n",
    "    \n",
    "    # Build the answer, adding semicolon only for pathogenic\n",
    "    if path == 'pathogenic' and diseases:\n",
    "        return f\"{path}; {disease_text}\"\n",
    "    else:\n",
    "        return path\n",
    "\n",
    "# Apply to your DataFrame\n",
    "training_df['answer'] = training_df.apply(format_answer, axis=1)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "disease_name\n",
       "not_provided                                                                                                      73241\n",
       "not_specified|not_provided                                                                                         6405\n",
       "not_provided|not_specified                                                                                         5466\n",
       "Inborn_genetic_diseases|not_provided                                                                               2289\n",
       "not_provided|Inborn_genetic_diseases                                                                               1929\n",
       "                                                                                                                  ...  \n",
       "not_provided|VAMP7-related_disorder                                                                                   1\n",
       "46,XY_sex_reversal_1|not_provided                                                                                     1\n",
       "Hereditary_factor_VIII_deficiency_disease|Thrombophilia,_X-linked,_due_to_factor_8_defect|not_provided                1\n",
       "Mendelian_susceptibility_to_mycobacterial_diseases_due_to_complete_ISG15_deficiency|not_specified|not_provided        1\n",
       "not_provided|not_specified|Mendelian_susceptibility_to_mycobacterial_diseases_due_to_complete_ISG15_deficiency        1\n",
       "Name: count, Length: 87193, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_df['disease_name'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in links: /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo2023/x86-64-v3, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo2023/generic, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic\n",
      "Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/networkx-3.4.2+computecanada-py3-none-any.whl\n",
      "Installing collected packages: networkx\n",
      "Successfully installed networkx-3.4.2+computecanada\n"
     ]
    }
   ],
   "source": [
    "!pip install --no-index networkx\n",
    "import networkx as nx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "disjoint diseases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting split assignment...\n",
      "Step 1/5: Building graph & disease→row index mapping (excluding specials)...\n",
      "  → Built graph with 13326 nodes, 48265 edges in 1.0s\n",
      "Step 2/5: Checking for existing disconnected components...\n",
      "  → Found 3099 components; skipping node removal.\n",
      "Step 5/5: Assigning rows to splits…\n",
      "Done! Total time: 9.8s; achieved train fraction = 0.1000\n",
      "Dropped diseases: []\n",
      "Rows dropped: 0\n",
      "Final train fraction: 0.100\n"
     ]
    }
   ],
   "source": [
    "special_diseases = {\"not_provided\", \"not_specified\", \"Inborn_genetic_diseases\", \"See_cases\"}\n",
    "import pandas as pd\n",
    "import networkx as nx\n",
    "import itertools\n",
    "import numpy as np\n",
    "import re\n",
    "import time\n",
    "from tqdm import tqdm\n",
    "from math import comb\n",
    "import multiprocessing as mp\n",
    "from functools import partial\n",
    "from collections import defaultdict\n",
    "\n",
    "def _evaluate_subset(subset, G, disease_to_rows, train_frac):\n",
    "    \"\"\"\n",
    "    Worker to evaluate one subset removal:\n",
    "      - removes `subset` from G,\n",
    "      - checks for ≥2 components,\n",
    "      - if so, computes the train/test split score using disease_to_rows.\n",
    "    Returns (score, subset, components) or None.\n",
    "    \"\"\"\n",
    "    H = G.copy()\n",
    "    H.remove_nodes_from(subset)\n",
    "    ccs = list(nx.connected_components(H))\n",
    "    if len(ccs) < 2:\n",
    "        return None\n",
    "\n",
    "    # compute unique row counts for each component\n",
    "    sizes = []\n",
    "    for comp in ccs:\n",
    "        rows = set()\n",
    "        for d in comp:\n",
    "            rows |= disease_to_rows.get(d, set())\n",
    "        sizes.append(len(rows))\n",
    "\n",
    "    # pick two largest comps\n",
    "    idx = np.argsort(sizes)[::-1][:2]\n",
    "    train_count, test_count = sizes[idx[0]], sizes[idx[1]]\n",
    "    frac = train_count / (train_count + test_count)\n",
    "    score = abs(frac - train_frac)\n",
    "    return (score, subset, ccs)\n",
    "\n",
    "def assign_disjoint_splits(\n",
    "    df: pd.DataFrame,\n",
    "    special_diseases: set,\n",
    "    train_frac: float = 0.9,\n",
    "    max_remove: int = 3,\n",
    "    random_state: int = 42,\n",
    "    n_procs: int = 24\n",
    ") -> (pd.DataFrame, dict):\n",
    "    \"\"\"\n",
    "    Add a 'split' column to df (0=train, 1=test) so that:\n",
    "      - No disease outside special_diseases appears in both splits.\n",
    "      - The overall train/test row ratio is as close to train_frac as possible.\n",
    "      - SNV/non-SNV and pathogenic/benign proportions stay balanced automatically\n",
    "        by sampling at the end for any rows containing only special diseases.\n",
    "    Uses up to `n_procs` parallel processes for the removal search, but only if needed.\n",
    "    Prints progress at every major step.\n",
    "    \"\"\"\n",
    "    rng = np.random.RandomState(random_state)\n",
    "    start_time = time.time()\n",
    "    print(\"Starting split assignment...\")\n",
    "\n",
    "    # 1) Build graph and disease→rows mapping\n",
    "    print(\"Step 1/5: Building graph & disease→row index mapping (excluding specials)...\")\n",
    "    G = nx.Graph()\n",
    "    disease_to_rows = defaultdict(set)\n",
    "    for idx, name_str in enumerate(df['disease_name']):\n",
    "        names = name_str.split('|')\n",
    "        non_special = [d for d in names if d not in special_diseases]\n",
    "        for d in non_special:\n",
    "            disease_to_rows[d].add(idx)\n",
    "            G.add_node(d)\n",
    "        for u, v in itertools.combinations(non_special, 2):\n",
    "            G.add_edge(u, v)\n",
    "    elapsed = time.time() - start_time\n",
    "    print(f\"  → Built graph with {G.number_of_nodes()} nodes, {G.number_of_edges()} edges in {elapsed:.1f}s\")\n",
    "\n",
    "    # 2) Check connectivity\n",
    "    print(\"Step 2/5: Checking for existing disconnected components...\")\n",
    "    comps = list(nx.connected_components(G))\n",
    "    if len(comps) >= 2:\n",
    "        print(f\"  → Found {len(comps)} components; skipping node removal.\")\n",
    "        # compute rows-per-component sets\n",
    "        comp_rows = []\n",
    "        for comp in comps:\n",
    "            rows_set = set()\n",
    "            for d in comp:\n",
    "                rows_set |= disease_to_rows[d]\n",
    "            comp_rows.append((comp, rows_set))\n",
    "\n",
    "        # total non-special rows\n",
    "        total_ns_rows = len(set().union(*(rows for _, rows in comp_rows)))\n",
    "        target_train_ns = train_frac * total_ns_rows\n",
    "\n",
    "        # sort components by descending size\n",
    "        comp_rows.sort(key=lambda x: len(x[1]), reverse=True)\n",
    "\n",
    "        # greedy pack to hit target_train_ns\n",
    "        train_comp = set()\n",
    "        train_rows = set()\n",
    "        for comp, rows_set in comp_rows:\n",
    "            if len(train_rows | rows_set) <= target_train_ns or not train_rows:\n",
    "                train_comp |= comp\n",
    "                train_rows |= rows_set\n",
    "\n",
    "        all_nodes = set(G.nodes())\n",
    "        test_comp = all_nodes - train_comp\n",
    "        dropped = []\n",
    "    else:\n",
    "        # 3) Removal search\n",
    "        print(\"Step 3/5: Graph is connected; searching for node removals…\")\n",
    "        best = {'score': float('inf')}\n",
    "        all_nodes = list(G.nodes())\n",
    "        worker = partial(_evaluate_subset,\n",
    "                         G=G,\n",
    "                         disease_to_rows=disease_to_rows,\n",
    "                         train_frac=train_frac)\n",
    "        for k in range(1, max_remove + 1):\n",
    "            total_combs = comb(len(all_nodes), k)\n",
    "            print(f\"  → Trying removals of size {k} ({total_combs} combos)…\")\n",
    "            with mp.Pool(processes=n_procs) as pool:\n",
    "                for result in tqdm(pool.imap_unordered(worker, itertools.combinations(all_nodes, k)),\n",
    "                                   total=total_combs,\n",
    "                                   desc=f\"    size={k}\"):\n",
    "                    if not result:\n",
    "                        continue\n",
    "                    score, subset, ccs = result\n",
    "                    if score < best['score']:\n",
    "                        best.update(score=score, subset=subset, components=ccs)\n",
    "            elapsed_k = time.time() - start_time\n",
    "            print(f\"    → Done size-{k} in {elapsed_k:.1f}s; best score = {best['score']:.4f}\")\n",
    "            if best['score'] < float('inf'):\n",
    "                break\n",
    "\n",
    "        dropped = list(best['subset'])\n",
    "        comps = best['components']\n",
    "\n",
    "        # 4) select two largest comps\n",
    "        print(\"Step 4/5: Selecting two largest components for train/test…\")\n",
    "        comp_counts = []\n",
    "        for comp in comps:\n",
    "            rows_set = set()\n",
    "            for d in comp:\n",
    "                rows_set |= disease_to_rows[d]\n",
    "            comp_counts.append((comp, rows_set))\n",
    "        comp_counts.sort(key=lambda x: len(x[1]), reverse=True)\n",
    "        train_comp, test_comp = comp_counts[0][0], comp_counts[1][0]\n",
    "\n",
    "    # 5) Assign rows\n",
    "    print(\"Step 5/5: Assigning rows to splits…\")\n",
    "    def which_split(dlist):\n",
    "        non_special = [d for d in dlist if d not in special_diseases]\n",
    "        if any(d in train_comp for d in non_special):\n",
    "            return 0\n",
    "        if any(d in test_comp for d in non_special):\n",
    "            return 1\n",
    "        return None\n",
    "\n",
    "    df_out = df.copy()\n",
    "    df_out['split'] = df_out['disease_name'].str.split('|').apply(which_split)\n",
    "\n",
    "    # fill None rows to achieve exact train_frac\n",
    "    mask_none = df_out['split'].isna()\n",
    "    n_none = mask_none.sum()\n",
    "    n_train_desired = int(train_frac * len(df_out))\n",
    "    n_current_train = (df_out['split'] == 0).sum()\n",
    "    n_to_train = max(0, n_train_desired - n_current_train)\n",
    "    assign = np.array([0]*n_to_train + [1]*(n_none - n_to_train))\n",
    "    rng.shuffle(assign)\n",
    "    df_out.loc[mask_none, 'split'] = assign\n",
    "    df_out['split'] = df_out['split'].astype(int)\n",
    "\n",
    "    total_elapsed = time.time() - start_time\n",
    "    print(f\"Done! Total time: {total_elapsed:.1f}s; achieved train fraction = {df_out['split'].mean():.4f}\")\n",
    "\n",
    "    info = {\n",
    "        'dropped_nodes': dropped,\n",
    "        'dropped_row_count': int(sum(len(disease_to_rows[d]) for d in dropped)),\n",
    "        'achieved_frac': float(df_out['split'].mean())\n",
    "    }\n",
    "    return df_out, info\n",
    "\n",
    "# ── Usage ──\n",
    "new_df, report = assign_disjoint_splits(\n",
    "    training_df,\n",
    "    special_diseases,\n",
    "    train_frac=0.9,\n",
    "    max_remove=3,\n",
    "    random_state=42,\n",
    "    n_procs=24\n",
    ")\n",
    "print(\"Dropped diseases:\", report['dropped_nodes'])\n",
    "print(\"Rows dropped:\", report['dropped_row_count'])\n",
    "print(f\"Final train fraction: {report['achieved_frac']:.3f}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "split\n",
       "0    308420\n",
       "1     34269\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df['split'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['SAMD11-related_disorder|not_provided',\n",
       "       'not_provided|SAMD11-related_disorder', 'not_provided', ...,\n",
       "       'not_provided|VAMP7-related_disorder',\n",
       "       '46,XY_sex_reversal_1|not_provided',\n",
       "       'TBL1Y-related_disorder|Deafness,_Y-linked_2|not_provided'],\n",
       "      shape=(11445,), dtype=object)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df[new_df['split']==1]['disease_name'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "split\n",
       "0    308420\n",
       "1     34269\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df['split'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== Split 0 (n=308420) ===\n",
      "\n",
      "Pathogenicity counts:\n",
      "cleaned_pathogenicity\n",
      "benign        230709\n",
      "pathogenic     77711\n",
      "\n",
      "Pathogenicity ratios:\n",
      "cleaned_pathogenicity\n",
      "benign        0.748035\n",
      "pathogenic    0.251965\n",
      "\n",
      "Variant-type counts:\n",
      "variant_type\n",
      "SNV        274147\n",
      "non_SNV     34273\n",
      "\n",
      "Variant-type ratios:\n",
      "variant_type\n",
      "SNV        0.888876\n",
      "non_SNV    0.111124\n",
      "\n",
      "=== Split 1 (n=34269) ===\n",
      "\n",
      "Pathogenicity counts:\n",
      "cleaned_pathogenicity\n",
      "benign        30279\n",
      "pathogenic     3990\n",
      "\n",
      "Pathogenicity ratios:\n",
      "cleaned_pathogenicity\n",
      "benign        0.883568\n",
      "pathogenic    0.116432\n",
      "\n",
      "Variant-type counts:\n",
      "variant_type\n",
      "SNV        32454\n",
      "non_SNV     1815\n",
      "\n",
      "Variant-type ratios:\n",
      "variant_type\n",
      "SNV        0.947037\n",
      "non_SNV    0.052963\n",
      "\n",
      "Cross-tab: split × pathogenicity\n",
      "cleaned_pathogenicity    benign  pathogenic\n",
      "split                                      \n",
      "0                      0.748035    0.251965\n",
      "1                      0.883568    0.116432\n",
      "\n",
      "Cross-tab: split × variant_type\n",
      "variant_type       SNV   non_SNV\n",
      "split                           \n",
      "0             0.888876  0.111124\n",
      "1             0.947037  0.052963\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# assuming new_df is your DataFrame with a 'split' column (0=train, 1=test)\n",
    "\n",
    "def print_ratio_stats(df, split_label):\n",
    "    sub = df[df['split'] == split_label]\n",
    "    total = len(sub)\n",
    "    print(f\"\\n=== Split {split_label} (n={total}) ===\")\n",
    "    \n",
    "    # Pathogenic vs. Benign\n",
    "    p_counts = sub['cleaned_pathogenicity'].value_counts()\n",
    "    p_ratios = p_counts / total\n",
    "    print(\"\\nPathogenicity counts:\")\n",
    "    print(p_counts.to_string())\n",
    "    print(\"\\nPathogenicity ratios:\")\n",
    "    print(p_ratios.to_string())\n",
    "    \n",
    "    # SNV vs. non-SNV\n",
    "    v_counts = sub['variant_type'].value_counts()\n",
    "    v_ratios = v_counts / total\n",
    "    print(\"\\nVariant-type counts:\")\n",
    "    print(v_counts.to_string())\n",
    "    print(\"\\nVariant-type ratios:\")\n",
    "    print(v_ratios.to_string())\n",
    "\n",
    "# Overall\n",
    "print_ratio_stats(new_df, 0)  # train\n",
    "print_ratio_stats(new_df, 1)  # test\n",
    "\n",
    "# If you also want a quick cross-tab view:\n",
    "print(\"\\nCross-tab: split × pathogenicity\")\n",
    "print(pd.crosstab(new_df['split'], new_df['cleaned_pathogenicity'], normalize='index'))\n",
    "\n",
    "print(\"\\nCross-tab: split × variant_type\")\n",
    "print(pd.crosstab(new_df['split'], new_df['variant_type'], normalize='index'))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "final_df = new_df.copy()[['question', 'answer', 'reference_sequence', 'mutated_sequence', 'split', 'variant_type', 'cleaned_pathogenicity']]\n",
    "\n",
    "# if len(final_df['variant_type'].value_counts().keys().tolist()) > 2:\n",
    "#     raise ValueError(\"variant_type has more than 2 values, should just be SNV and non_SNV\")\n",
    "\n",
    "train_split_df = final_df[final_df['split']==0]\n",
    "test_split_df = final_df[final_df['split']==1]\n",
    "\n",
    "train_split_df = train_split_df.drop('split', axis=1)\n",
    "test_split_df = test_split_df.drop('split', axis=1)\n",
    "\n",
    "snv_train_split_df = train_split_df[train_split_df['variant_type']=='SNV']\n",
    "non_snv_train_split_df = train_split_df[train_split_df['variant_type']=='non_SNV']\n",
    "\n",
    "snv_test_split_df = test_split_df[test_split_df['variant_type']=='SNV']\n",
    "non_snv_test_split_df = test_split_df[test_split_df['variant_type']=='non_SNV']\n",
    "\n",
    "snv_test_split_df = snv_test_split_df.drop('variant_type', axis=1)\n",
    "non_snv_test_split_df = non_snv_test_split_df.drop('variant_type', axis=1)\n",
    "\n",
    "snv_train_split_df = snv_train_split_df.drop('variant_type', axis=1)\n",
    "non_snv_train_split_df = non_snv_train_split_df.drop('variant_type', axis=1)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "      <th>reference_sequence</th>\n",
       "      <th>mutated_sequence</th>\n",
       "      <th>cleaned_pathogenicity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Assess the variant on chromosome 1, position 9...</td>\n",
       "      <td>benign</td>\n",
       "      <td>GGCCGAGGCGGGCGGATCACGAGGTCAGGAGATCGAGACCATCCTG...</td>\n",
       "      <td>GGCCGAGGCGGGCGGATCACGAGGTCAGGAGATCGAGACCATCCTG...</td>\n",
       "      <td>benign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Gene SAMD11 (sterile alpha motif domain contai...</td>\n",
       "      <td>benign</td>\n",
       "      <td>TGACTAACACGGTGAAACCCGTCTCTACTAAAAATACAAAAAATTA...</td>\n",
       "      <td>TGACTAACACGGTGAAACCCGTCTCTACTAAAAATACAAAAAATTA...</td>\n",
       "      <td>benign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The mutation in gene SAMD11 (sterile alpha mot...</td>\n",
       "      <td>benign</td>\n",
       "      <td>CCTGTAGTCCCAGCTACTTGGGAGGCTGAGGCAGGAGAATGGCATG...</td>\n",
       "      <td>CCTGTAGTCCCAGCTACTTGGGAGGCTGAGGCAGGAGAATGGCATG...</td>\n",
       "      <td>benign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Determine whether the variant at chromosome 1,...</td>\n",
       "      <td>benign</td>\n",
       "      <td>GAGGGAGTGAGTTAGACGCTCTCAAGGGCTCTGCCACCTCCCGGAG...</td>\n",
       "      <td>GAGGGAGTGAGTTAGACGCTCTCAAGGGCTCTGCCACCTCCCGGAG...</td>\n",
       "      <td>benign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Variant on chromosome 1, at position 935779, a...</td>\n",
       "      <td>benign</td>\n",
       "      <td>CCTATGTGCCTGGGGGGGGCTTCCTTTCCCACTGGGAGCCGGTGGG...</td>\n",
       "      <td>CCTATGTGCCTGGGGGGGGCTTCCTTTCCCACTGGGAGCCGGTGGG...</td>\n",
       "      <td>benign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342678</th>\n",
       "      <td>Variant at chromosome X, position 155524483, g...</td>\n",
       "      <td>benign</td>\n",
       "      <td>GTGTGCATAGCTCTATGCAGTGTAATTACATGTGTAACTTTGTGTA...</td>\n",
       "      <td>GTGTGCATAGCTCTATGCAGTGTAATTACATGTGTAACTTTGTGTA...</td>\n",
       "      <td>benign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342680</th>\n",
       "      <td>Mutation at chromosome X, position 155900534, ...</td>\n",
       "      <td>benign</td>\n",
       "      <td>AGCATTAAAGATCATCTAGTTGAACTACCCATCTGATGCTTAAATG...</td>\n",
       "      <td>AGCATTAAAGATCATCTAGTTGAACTACCCATCTGATGCTTAAATG...</td>\n",
       "      <td>benign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342681</th>\n",
       "      <td>Does the variant on chromosome X at location 1...</td>\n",
       "      <td>benign</td>\n",
       "      <td>CAATTAGTCCCTTGATTATTGATCCTTCTCTTTTGGCTGTATTCTC...</td>\n",
       "      <td>CAATTAGTCCCTTGATTATTGATCCTTCTCTTTTGGCTGTATTCTC...</td>\n",
       "      <td>benign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342685</th>\n",
       "      <td>Assess the clinical significance (benign or pa...</td>\n",
       "      <td>benign</td>\n",
       "      <td>TTTAGTCTTTCCAAAATGTATACATGCATGATGTCATAATTTTTAA...</td>\n",
       "      <td>TTTAGTCTTTCCAAAATGTATACATGCATGATGTCATAATTTTTAA...</td>\n",
       "      <td>benign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342686</th>\n",
       "      <td>Is the variant located on chromosome Y at posi...</td>\n",
       "      <td>benign</td>\n",
       "      <td>AGGTGGCCGTGGCTGTCTGAGGGGAAAGACTGGGGACACTGAATGG...</td>\n",
       "      <td>AGGTGGCCGTGGCTGTCTGAGGGGAAAGACTGGGGACACTGAATGG...</td>\n",
       "      <td>benign</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>32454 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 question  answer  \\\n",
       "0       Assess the variant on chromosome 1, position 9...  benign   \n",
       "1       Gene SAMD11 (sterile alpha motif domain contai...  benign   \n",
       "2       The mutation in gene SAMD11 (sterile alpha mot...  benign   \n",
       "3       Determine whether the variant at chromosome 1,...  benign   \n",
       "4       Variant on chromosome 1, at position 935779, a...  benign   \n",
       "...                                                   ...     ...   \n",
       "342678  Variant at chromosome X, position 155524483, g...  benign   \n",
       "342680  Mutation at chromosome X, position 155900534, ...  benign   \n",
       "342681  Does the variant on chromosome X at location 1...  benign   \n",
       "342685  Assess the clinical significance (benign or pa...  benign   \n",
       "342686  Is the variant located on chromosome Y at posi...  benign   \n",
       "\n",
       "                                       reference_sequence  \\\n",
       "0       GGCCGAGGCGGGCGGATCACGAGGTCAGGAGATCGAGACCATCCTG...   \n",
       "1       TGACTAACACGGTGAAACCCGTCTCTACTAAAAATACAAAAAATTA...   \n",
       "2       CCTGTAGTCCCAGCTACTTGGGAGGCTGAGGCAGGAGAATGGCATG...   \n",
       "3       GAGGGAGTGAGTTAGACGCTCTCAAGGGCTCTGCCACCTCCCGGAG...   \n",
       "4       CCTATGTGCCTGGGGGGGGCTTCCTTTCCCACTGGGAGCCGGTGGG...   \n",
       "...                                                   ...   \n",
       "342678  GTGTGCATAGCTCTATGCAGTGTAATTACATGTGTAACTTTGTGTA...   \n",
       "342680  AGCATTAAAGATCATCTAGTTGAACTACCCATCTGATGCTTAAATG...   \n",
       "342681  CAATTAGTCCCTTGATTATTGATCCTTCTCTTTTGGCTGTATTCTC...   \n",
       "342685  TTTAGTCTTTCCAAAATGTATACATGCATGATGTCATAATTTTTAA...   \n",
       "342686  AGGTGGCCGTGGCTGTCTGAGGGGAAAGACTGGGGACACTGAATGG...   \n",
       "\n",
       "                                         mutated_sequence  \\\n",
       "0       GGCCGAGGCGGGCGGATCACGAGGTCAGGAGATCGAGACCATCCTG...   \n",
       "1       TGACTAACACGGTGAAACCCGTCTCTACTAAAAATACAAAAAATTA...   \n",
       "2       CCTGTAGTCCCAGCTACTTGGGAGGCTGAGGCAGGAGAATGGCATG...   \n",
       "3       GAGGGAGTGAGTTAGACGCTCTCAAGGGCTCTGCCACCTCCCGGAG...   \n",
       "4       CCTATGTGCCTGGGGGGGGCTTCCTTTCCCACTGGGAGCCGGTGGG...   \n",
       "...                                                   ...   \n",
       "342678  GTGTGCATAGCTCTATGCAGTGTAATTACATGTGTAACTTTGTGTA...   \n",
       "342680  AGCATTAAAGATCATCTAGTTGAACTACCCATCTGATGCTTAAATG...   \n",
       "342681  CAATTAGTCCCTTGATTATTGATCCTTCTCTTTTGGCTGTATTCTC...   \n",
       "342685  TTTAGTCTTTCCAAAATGTATACATGCATGATGTCATAATTTTTAA...   \n",
       "342686  AGGTGGCCGTGGCTGTCTGAGGGGAAAGACTGGGGACACTGAATGG...   \n",
       "\n",
       "       cleaned_pathogenicity  \n",
       "0                     benign  \n",
       "1                     benign  \n",
       "2                     benign  \n",
       "3                     benign  \n",
       "4                     benign  \n",
       "...                      ...  \n",
       "342678                benign  \n",
       "342680                benign  \n",
       "342681                benign  \n",
       "342685                benign  \n",
       "342686                benign  \n",
       "\n",
       "[32454 rows x 5 columns]"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# save all the final dataframes to parquet files\n",
    "snv_train_split_df.to_parquet('SCRATCH_DIR/DNASNVData113/finaldata/snv_train_split_df.parquet')\n",
    "non_snv_train_split_df.to_parquet('SCRATCH_DIR/DNASNVData113/finaldata/non_snv_train_split_df.parquet')\n",
    "snv_test_split_df.to_parquet('SCRATCH_DIR/DNASNVData113/finaldata/snv_test_split_df.parquet')\n",
    "non_snv_test_split_df.to_parquet('SCRATCH_DIR/DNASNVData113/finaldata/non_snv_test_split_df.parquet')\n",
    "\n",
    "#now upload to huggingface\n",
    "!pip install --no-index huggingface-hub\n",
    "from huggingface_hub import HfApi\n",
    "import os\n",
    "import glob\n",
    "\n",
    "# 0) config\n",
    "repo_id     = \"wanglab/bioR_tasks\"         # your dataset repo\n",
    "repo_type   = \"dataset\"\n",
    "subfolder   = \"task4-variant_effect_non_snv_and_snv_with_split\"\n",
    "local_dir   = \"SCRATCH_DIR/DNASNVData113/clinvar_data/clinvar_windowed_4096.parquet\"\n",
    "\n",
    "api = HfApi()\n",
    "\n",
    "# 1) list all files in that subfolder\n",
    "all_files = api.list_repo_files(repo_id, repo_type=repo_type)\n",
    "old_files = [f for f in all_files if f.startswith(subfolder + \"/\")]\n",
    "\n",
    "print(f\"Will delete {len(old_files)} old files:\")\n",
    "for f in old_files:\n",
    "    print(\"  \", f)\n",
    "\n",
    "# 2) delete them (one commit per file, or you can batch by reusing the same commit_message)\n",
    "for f in old_files:\n",
    "    api.delete_file(\n",
    "        path_in_repo = f,\n",
    "        repo_id      = repo_id,\n",
    "        repo_type    = repo_type,\n",
    "        commit_message = f\"remove old dataset file\"\n",
    "    )\n",
    "\n",
    "# 3) upload your single Parquet file\n",
    "new_file = \"SCRATCH_DIR/DNASNVData113/clinvar_data/clinvar_windowed_4096.parquet\"\n",
    "basename = os.path.basename(new_file)\n",
    "dest_path = f\"{subfolder}/{basename}\"\n",
    "\n",
    "print(f\"Uploading {new_file!r} to {repo_id}/{dest_path} …\")\n",
    "api.upload_file(\n",
    "    path_or_fileobj = new_file,\n",
    "    path_in_repo    = dest_path,\n",
    "    repo_id         = repo_id,\n",
    "    repo_type       = repo_type,\n",
    "    commit_message  = f\"add updated parquet {basename}\"\n",
    ")\n",
    "\n",
    "print(\"Done! Your dataset has been updated on the Hub.\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'question': 'Assess the variant on chromosome 1, position 930204, impacting SAMD11 (sterile alpha motif domain containing 11): is it benign or pathogenic? If pathogenic, specify the associated condition(s).',\n",
       " 'answer': 'benign',\n",
       " 'reference_sequence': 'GGCCGAGGCGGGCGGATCACGAGGTCAGGAGATCGAGACCATCCTGACTAACACGGTGAAACCCGTCTCTACTAAAAATACAAAAAATTAGCCGGGCGTGGTGGCGGGTGCCTGTAGTCCCAGCTACTTGGGAGGCTGAGGCAGGAGAATGGCATGAACCCGGGAGGCGGAGCTTGCAGTGAGCCCAGATTGTGCCACCGCACTCCAGCCTGGGCAACAGAGTGAGACTCCGTCTCAAAAAACTAAAAAAGAAGAGAGGTGGGAGAGGAGAGGCTGTCAGAGCCTCTAAGCCCTGGTGCTTGGGCTGCAGAAGGGCAGAGCTAAGCGGGACTTCCCAGCACAGCACACTCCGGACAGGCTGTGGCTGTTGAAGGGACCCCCGAGCTCCAGCTGACACGCGGAGGCCCGGGCACAGACAGGCATCATACCTTCGGCCTTGGCCGCACTCTGTGGTCATTGGTGTTGGGGGCAGCCCAGGGTCAGGGCAGGGTCTCAGCCTCGGACCCCAGGCCCCACCCCTTGCCCAGCAGTGCTGCGTTTTCCCAGTGAGCTGTCGTGGAGAGAGCAGAGGGGACCCAGCGCAGGCCCAGTGGCCGGTGAGGGGAGACGTGGCTCTGGGACGGGGGCCTCCACCTGGGTGGGGGGATGCTCCAGCTTCCAGACCCTTGGGGAGGGGGCACTGCCCAAACTAAGCTGGCACTGGGGCTGTGCATTTGAAGGTGATGGTGGTTCTAGGTCTGAGGAGGACACCCTCCTAACAGCCTCATCCCCAAGCTCCGGGCTGTGTTGTGGCAATGGGAGGGAGGAAGTCTGAGGAGACCCTGGTGACTGAACGGAGGAGGGAGTGAGTTAGACGCTCTCAAGGGCTCTGCCACCTCCCGGAGCCAGCGGCCTGTTACTACATTTAAAAAAGCCTCCCGCCCACTGGAAAATAATCAATAACTTTCCTTTATCCCTGGGGGTGGCAGGACCTAGAAACACTGGAGGAGTCCGGAAGTGCCTGGGGCTGGGCCGGCGCTGGTGTGCTGTGCAGGGTGCCGCGGGCACGTCCGCCGCGTGTGTGCGTCAGCTCGGGGCTCGGCTGTGCTCTGCAGGGACCACAGCGGGCGTGTCTGTGCTCCCACCCGAGGCACCCACAGCTCCACACGCTCGTTCCGTGGGTGCAAAGGAGATGGGAGAAAGAAGCCCTGTGAGAAATGCGGGGCAGGGTTTGCGGAACAGGGGACCTGGGCTGGTGAGGGCTCCTCGTCTGGTGACCTGTGAGCCCCGGGGCCTGCAGTCTGCGAGGGTTCAGCTCAGACAGTTGCCAGTGGCCTTGCACCAGGCTGCAGCTGCCCCTGAGCCGGGCTGTGCGTGGCGCTGATGAAATAGAAAAGGGCATTCGCTTGTCAACGTTGGCATCGGTGGCAGGGTGTGGTGGGCAGAAGGGTCACAAAGTACGGGTGGGATTGGCAGGCAGATACACGGAGGGAACGTGCGCATTTGAGTGCACGTCCACCAGCACCAGCCCCAGGCCACAGGCAGATCCCAGGAGACACGCAGGGGCCCTAAGAAGGGAGCTGGGAATGAGGGGCCACACAAGCCCGGGACGGAGGCCTGTCGCACATGGGGTGGCCCCGACTCAGGCCCTGGAGTTGGCCAGGACCCTCTAGCATCCTCAAGGGCTGGGCCAACCAGGCTGGCGTGGGGTGGGGCAGGGGAGGGCTGAGCCAGTGGGCGTCGTCTGTAGGGGGATGCCCAACTGCGGCCCCGTCTCTCGGCTCTCCTCTGGGTCTCTGGCCAGCTGTGGCTCCTGCTGGCCCCAGGCGCATCCCAGAGGCAGGTAGAGGGAGGATGGCTGCTCTGAGGGCACCTCTGCCGTGCTTGGGGCTCGGCCTGGGGTGCGAGACCAGGGCAGACCCCCGGGAGATGGAACGGCCCGGTCCAGCCCCACCTTCCTCTCCTCCTGCCCCACCTTCCTCTCCTCCTGCCCCACCTTCCTCTCCTCCTGCCCCACCAGAACCGGGGGCGGCTGGCAGACAAGAGGACAGTCGCCCTGCCTGCCGCCCGGAACCTGAAGAAGGAGCGAACTCCCAGCTTCTCTGCCAGCGATGGTGACAGCGACGGGAGTGGCCCCACCTGTGGGCGGCGGCCAGGCTTGAAGCAGGAGGATGGTCCGCACATCCGTATCATGAAGAGAAGGTACTTGGACCAGGGCCGGACAGGAAGGCGCAAGGCTCAGATGGGGCTGGAGCTTCAGGCCTTCAGCTGCTCAGATGAGAGTGTCCACACCGGCCTCCCACACCTTCCCTCAGATGCTGGTCTTTTTGGGGTCCTGTGTGGGTCGCAGGCAGGAGCTGTTTCCTCATCTGCCCCCTGTCTGGCGTCCCCTCCCACCTCTGCTCTGCGGCGCTCACTGGCAGAGGCAGGTTGGCAGCAGTTGGGACCCAGAGGTCTGCACCTTCCTGGGCCGACGCTCCAGCTACCCTTGCTGACCGGGTCCCAGTCTGGCCAGAGAGCAGCTCTAGCAACAGGGAGCTCCATTCAGGCTCGTGACTGGCTGTGCAGAAGCAGCCTCGGCCCCCACCTGCGGTACAACAGGAGGGCTCCTCTGAGTGCACGGCAACAAGCAAGAGGGAGAAGGGGCCTCGGTCCTGTTCTTCCTGATGCGTGTCTGCTGAGGCCAGGAGCTGGCTTTGGCCCATGGGCCTGTCCTAGTGGGAGGCCCCAGCATGTTGAGCCAGTAGCAGGTGGTGCTGGGCATGGCAGCCGCCCTCGTTCACTGCCCAGGGCTGTGGCCCAGCGGGGCACTGACCCGAGACAGGTCTGCGCACGCCCTGCTATCCTGAGGCTGGGGTCAGGGGCCTCCAGAGCAACATGGACCTTCTGCTTCCCTTCCTGCAGAGTCCACACCCACTGGGACGTGAACATCTCTTTCCGAGAGGCGTCCTGCAGGTAGGAGCCGTGCTGTGCGTGCATAAGAGGGGGCCGTGACTCCCCTCCCTCCCTCCCACCCCTGACCGTGCCCTGCTGTCTGCTGTCCGCTGTCTCAGCGTGAGCTGATGCTGTGATGCTGGCTGAGTGTCTGCCAGGTTTGACATGTGCTGCAAGGTTGTCCCCCATCCCGGGAGGCAGACAGTGTTGCACCCAGTTGGGACTGAGGGACCCCAGACCCAGTCAGATGCAGCTCTCGGCAGCAGCTCAGGTGTGAGTTCTGGGCAGCCCGGCCCTGGAGTTAGAGTGCACTTCCTCCCATGTGAGACTGGCCATTTGAGCCCAAAAATGAGGCTGTCACCTCCCCCTTCCCACCCTCCTAGAGACCCACAAGGAGGTGAGAATGCTGATGTGTGAGTGGGGCCCTGAAGGGTGTGTAGGAGCTCTAAGGCGAGGGGATGTCTGCAGAGTAGAGGAACAGGGAAGGGCGTGTAGGAGGGACGAGGAGTGAACCTGGCAGCTCTGGTTCAGTTGGATGCTGAAGAGTCATGGATGCTGGGCCTGTGGGCACCGTCCTCCAGGCGGGAGCCACCGAAAGTTCTTGAGCAGGGCAGTGACCAGGTGTATGTTTGGAGAAGGTCCCTCTGGAGGCCTTCCTGGCAGACAGGGGATTGGATTCAGGCTGTGGAAGCAGGACGGTAGGGGGTGTGATTCCAGGATGTGGAAAGGAGATAAAAATGAAGAGCCCCGGGGAAGAGGTCAAGGGAGTTGGGGGACCCGAGTTCCTGGCTCCAGGGGGAAGCGAGTGGTAAGTCTGTGAACAGAGCCCAGCTGTGGATTCTGTCAATGGGGTCAGGTCTCACCCTGTGGCTTCCAGGGCAGCAAGGCAGGAAGGAGGCGTCTGCCACAAGGCCAGCTTCCTGGGGCCAGAGCCGTGAAGGCCCAGGGGACCTGCGTGTCTTGGCTCCACGCCAGATGTGTTATTATTTATGTCTCTGAGAATGTCTGGATCTCAGAGCCGAATTACAATAAAAACATCTTTAAACTTATTTCTACCTCATTTTGGGGTTGCCAGCTCACCTGATCATTTTTATGAACTGTCATGAACACTGATGACATTTTATGAGCCTTTTACATGGGACACTACAGAATACATTTGTCAGCGAGGCCTGTAGGGAAACCC',\n",
       " 'mutated_sequence': 'GGCCGAGGCGGGCGGATCACGAGGTCAGGAGATCGAGACCATCCTGACTAACACGGTGAAACCCGTCTCTACTAAAAATACAAAAAATTAGCCGGGCGTGGTGGCGGGTGCCTGTAGTCCCAGCTACTTGGGAGGCTGAGGCAGGAGAATGGCATGAACCCGGGAGGCGGAGCTTGCAGTGAGCCCAGATTGTGCCACCGCACTCCAGCCTGGGCAACAGAGTGAGACTCCGTCTCAAAAAACTAAAAAAGAAGAGAGGTGGGAGAGGAGAGGCTGTCAGAGCCTCTAAGCCCTGGTGCTTGGGCTGCAGAAGGGCAGAGCTAAGCGGGACTTCCCAGCACAGCACACTCCGGACAGGCTGTGGCTGTTGAAGGGACCCCCGAGCTCCAGCTGACACGCGGAGGCCCGGGCACAGACAGGCATCATACCTTCGGCCTTGGCCGCACTCTGTGGTCATTGGTGTTGGGGGCAGCCCAGGGTCAGGGCAGGGTCTCAGCCTCGGACCCCAGGCCCCACCCCTTGCCCAGCAGTGCTGCGTTTTCCCAGTGAGCTGTCGTGGAGAGAGCAGAGGGGACCCAGCGCAGGCCCAGTGGCCGGTGAGGGGAGACGTGGCTCTGGGACGGGGGCCTCCACCTGGGTGGGGGGATGCTCCAGCTTCCAGACCCTTGGGGAGGGGGCACTGCCCAAACTAAGCTGGCACTGGGGCTGTGCATTTGAAGGTGATGGTGGTTCTAGGTCTGAGGAGGACACCCTCCTAACAGCCTCATCCCCAAGCTCCGGGCTGTGTTGTGGCAATGGGAGGGAGGAAGTCTGAGGAGACCCTGGTGACTGAACGGAGGAGGGAGTGAGTTAGACGCTCTCAAGGGCTCTGCCACCTCCCGGAGCCAGCGGCCTGTTACTACATTTAAAAAAGCCTCCCGCCCACTGGAAAATAATCAATAACTTTCCTTTATCCCTGGGGGTGGCAGGACCTAGAAACACTGGAGGAGTCCGGAAGTGCCTGGGGCTGGGCCGGCGCTGGTGTGCTGTGCAGGGTGCCGCGGGCACGTCCGCCGCGTGTGTGCGTCAGCTCGGGGCTCGGCTGTGCTCTGCAGGGACCACAGCGGGCGTGTCTGTGCTCCCACCCGAGGCACCCACAGCTCCACACGCTCGTTCCGTGGGTGCAAAGGAGATGGGAGAAAGAAGCCCTGTGAGAAATGCGGGGCAGGGTTTGCGGAACAGGGGACCTGGGCTGGTGAGGGCTCCTCGTCTGGTGACCTGTGAGCCCCGGGGCCTGCAGTCTGCGAGGGTTCAGCTCAGACAGTTGCCAGTGGCCTTGCACCAGGCTGCAGCTGCCCCTGAGCCGGGCTGTGCGTGGCGCTGATGAAATAGAAAAGGGCATTCGCTTGTCAACGTTGGCATCGGTGGCAGGGTGTGGTGGGCAGAAGGGTCACAAAGTACGGGTGGGATTGGCAGGCAGATACACGGAGGGAACGTGCGCATTTGAGTGCACGTCCACCAGCACCAGCCCCAGGCCACAGGCAGATCCCAGGAGACACGCAGGGGCCCTAAGAAGGGAGCTGGGAATGAGGGGCCACACAAGCCCGGGACGGAGGCCTGTCGCACATGGGGTGGCCCCGACTCAGGCCCTGGAGTTGGCCAGGACCCTCTAGCATCCTCAAGGGCTGGGCCAACCAGGCTGGCGTGGGGTGGGGCAGGGGAGGGCTGAGCCAGTGGGCGTCGTCTGTAGGGGGATGCCCAACTGCGGCCCCGTCTCTCGGCTCTCCTCTGGGTCTCTGGCCAGCTGTGGCTCCTGCTGGCCCCAGGCGCATCCCAGAGGCAGGTAGAGGGAGGATGGCTGCTCTGAGGGCACCTCTGCCGTGCTTGGGGCTCGGCCTGGGGTGCGAGACCAGGGCAGACCCCCGGGAGATGGAACGGCCCGGTCCAGCCCCACCTTCCTCTCCTCCTGCCCCACCTTCCTCTCCTCCTGCCCCACCTTCCTCTCCTCCTGCCCCACCAGAACCGGGGGCGGCTGGCAGACAAGAGGACAGTCGCCCTGCCTGCCGCCCAGAACCTGAAGAAGGAGCGAACTCCCAGCTTCTCTGCCAGCGATGGTGACAGCGACGGGAGTGGCCCCACCTGTGGGCGGCGGCCAGGCTTGAAGCAGGAGGATGGTCCGCACATCCGTATCATGAAGAGAAGGTACTTGGACCAGGGCCGGACAGGAAGGCGCAAGGCTCAGATGGGGCTGGAGCTTCAGGCCTTCAGCTGCTCAGATGAGAGTGTCCACACCGGCCTCCCACACCTTCCCTCAGATGCTGGTCTTTTTGGGGTCCTGTGTGGGTCGCAGGCAGGAGCTGTTTCCTCATCTGCCCCCTGTCTGGCGTCCCCTCCCACCTCTGCTCTGCGGCGCTCACTGGCAGAGGCAGGTTGGCAGCAGTTGGGACCCAGAGGTCTGCACCTTCCTGGGCCGACGCTCCAGCTACCCTTGCTGACCGGGTCCCAGTCTGGCCAGAGAGCAGCTCTAGCAACAGGGAGCTCCATTCAGGCTCGTGACTGGCTGTGCAGAAGCAGCCTCGGCCCCCACCTGCGGTACAACAGGAGGGCTCCTCTGAGTGCACGGCAACAAGCAAGAGGGAGAAGGGGCCTCGGTCCTGTTCTTCCTGATGCGTGTCTGCTGAGGCCAGGAGCTGGCTTTGGCCCATGGGCCTGTCCTAGTGGGAGGCCCCAGCATGTTGAGCCAGTAGCAGGTGGTGCTGGGCATGGCAGCCGCCCTCGTTCACTGCCCAGGGCTGTGGCCCAGCGGGGCACTGACCCGAGACAGGTCTGCGCACGCCCTGCTATCCTGAGGCTGGGGTCAGGGGCCTCCAGAGCAACATGGACCTTCTGCTTCCCTTCCTGCAGAGTCCACACCCACTGGGACGTGAACATCTCTTTCCGAGAGGCGTCCTGCAGGTAGGAGCCGTGCTGTGCGTGCATAAGAGGGGGCCGTGACTCCCCTCCCTCCCTCCCACCCCTGACCGTGCCCTGCTGTCTGCTGTCCGCTGTCTCAGCGTGAGCTGATGCTGTGATGCTGGCTGAGTGTCTGCCAGGTTTGACATGTGCTGCAAGGTTGTCCCCCATCCCGGGAGGCAGACAGTGTTGCACCCAGTTGGGACTGAGGGACCCCAGACCCAGTCAGATGCAGCTCTCGGCAGCAGCTCAGGTGTGAGTTCTGGGCAGCCCGGCCCTGGAGTTAGAGTGCACTTCCTCCCATGTGAGACTGGCCATTTGAGCCCAAAAATGAGGCTGTCACCTCCCCCTTCCCACCCTCCTAGAGACCCACAAGGAGGTGAGAATGCTGATGTGTGAGTGGGGCCCTGAAGGGTGTGTAGGAGCTCTAAGGCGAGGGGATGTCTGCAGAGTAGAGGAACAGGGAAGGGCGTGTAGGAGGGACGAGGAGTGAACCTGGCAGCTCTGGTTCAGTTGGATGCTGAAGAGTCATGGATGCTGGGCCTGTGGGCACCGTCCTCCAGGCGGGAGCCACCGAAAGTTCTTGAGCAGGGCAGTGACCAGGTGTATGTTTGGAGAAGGTCCCTCTGGAGGCCTTCCTGGCAGACAGGGGATTGGATTCAGGCTGTGGAAGCAGGACGGTAGGGGGTGTGATTCCAGGATGTGGAAAGGAGATAAAAATGAAGAGCCCCGGGGAAGAGGTCAAGGGAGTTGGGGGACCCGAGTTCCTGGCTCCAGGGGGAAGCGAGTGGTAAGTCTGTGAACAGAGCCCAGCTGTGGATTCTGTCAATGGGGTCAGGTCTCACCCTGTGGCTTCCAGGGCAGCAAGGCAGGAAGGAGGCGTCTGCCACAAGGCCAGCTTCCTGGGGCCAGAGCCGTGAAGGCCCAGGGGACCTGCGTGTCTTGGCTCCACGCCAGATGTGTTATTATTTATGTCTCTGAGAATGTCTGGATCTCAGAGCCGAATTACAATAAAAACATCTTTAAACTTATTTCTACCTCATTTTGGGGTTGCCAGCTCACCTGATCATTTTTATGAACTGTCATGAACACTGATGACATTTTATGAGCCTTTTACATGGGACACTACAGAATACATTTGTCAGCGAGGCCTGTAGGGAAACCC'}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_training_df.iloc[0][['question', 'answer', 'reference_sequence', 'mutated_sequence']].to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'clinvar_id': '1170208',\n",
       " 'original_window': 'GGCCGAGGCGGGCGGATCACGAGGTCAGGAGATCGAGACCATCCTGACTAACACGGTGAAACCCGTCTCTACTAAAAATACAAAAAATTAGCCGGGCGTGGTGGCGGGTGCCTGTAGTCCCAGCTACTTGGGAGGCTGAGGCAGGAGAATGGCATGAACCCGGGAGGCGGAGCTTGCAGTGAGCCCAGATTGTGCCACCGCACTCCAGCCTGGGCAACAGAGTGAGACTCCGTCTCAAAAAACTAAAAAAGAAGAGAGGTGGGAGAGGAGAGGCTGTCAGAGCCTCTAAGCCCTGGTGCTTGGGCTGCAGAAGGGCAGAGCTAAGCGGGACTTCCCAGCACAGCACACTCCGGACAGGCTGTGGCTGTTGAAGGGACCCCCGAGCTCCAGCTGACACGCGGAGGCCCGGGCACAGACAGGCATCATACCTTCGGCCTTGGCCGCACTCTGTGGTCATTGGTGTTGGGGGCAGCCCAGGGTCAGGGCAGGGTCTCAGCCTCGGACCCCAGGCCCCACCCCTTGCCCAGCAGTGCTGCGTTTTCCCAGTGAGCTGTCGTGGAGAGAGCAGAGGGGACCCAGCGCAGGCCCAGTGGCCGGTGAGGGGAGACGTGGCTCTGGGACGGGGGCCTCCACCTGGGTGGGGGGATGCTCCAGCTTCCAGACCCTTGGGGAGGGGGCACTGCCCAAACTAAGCTGGCACTGGGGCTGTGCATTTGAAGGTGATGGTGGTTCTAGGTCTGAGGAGGACACCCTCCTAACAGCCTCATCCCCAAGCTCCGGGCTGTGTTGTGGCAATGGGAGGGAGGAAGTCTGAGGAGACCCTGGTGACTGAACGGAGGAGGGAGTGAGTTAGACGCTCTCAAGGGCTCTGCCACCTCCCGGAGCCAGCGGCCTGTTACTACATTTAAAAAAGCCTCCCGCCCACTGGAAAATAATCAATAACTTTCCTTTATCCCTGGGGGTGGCAGGACCTAGAAACACTGGAGGAGTCCGGAAGTGCCTGGGGCTGGGCCGGCGCTGGTGTGCTGTGCAGGGTGCCGCGGGCACGTCCGCCGCGTGTGTGCGTCAGCTCGGGGCTCGGCTGTGCTCTGCAGGGACCACAGCGGGCGTGTCTGTGCTCCCACCCGAGGCACCCACAGCTCCACACGCTCGTTCCGTGGGTGCAAAGGAGATGGGAGAAAGAAGCCCTGTGAGAAATGCGGGGCAGGGTTTGCGGAACAGGGGACCTGGGCTGGTGAGGGCTCCTCGTCTGGTGACCTGTGAGCCCCGGGGCCTGCAGTCTGCGAGGGTTCAGCTCAGACAGTTGCCAGTGGCCTTGCACCAGGCTGCAGCTGCCCCTGAGCCGGGCTGTGCGTGGCGCTGATGAAATAGAAAAGGGCATTCGCTTGTCAACGTTGGCATCGGTGGCAGGGTGTGGTGGGCAGAAGGGTCACAAAGTACGGGTGGGATTGGCAGGCAGATACACGGAGGGAACGTGCGCATTTGAGTGCACGTCCACCAGCACCAGCCCCAGGCCACAGGCAGATCCCAGGAGACACGCAGGGGCCCTAAGAAGGGAGCTGGGAATGAGGGGCCACACAAGCCCGGGACGGAGGCCTGTCGCACATGGGGTGGCCCCGACTCAGGCCCTGGAGTTGGCCAGGACCCTCTAGCATCCTCAAGGGCTGGGCCAACCAGGCTGGCGTGGGGTGGGGCAGGGGAGGGCTGAGCCAGTGGGCGTCGTCTGTAGGGGGATGCCCAACTGCGGCCCCGTCTCTCGGCTCTCCTCTGGGTCTCTGGCCAGCTGTGGCTCCTGCTGGCCCCAGGCGCATCCCAGAGGCAGGTAGAGGGAGGATGGCTGCTCTGAGGGCACCTCTGCCGTGCTTGGGGCTCGGCCTGGGGTGCGAGACCAGGGCAGACCCCCGGGAGATGGAACGGCCCGGTCCAGCCCCACCTTCCTCTCCTCCTGCCCCACCTTCCTCTCCTCCTGCCCCACCTTCCTCTCCTCCTGCCCCACCAGAACCGGGGGCGGCTGGCAGACAAGAGGACAGTCGCCCTGCCTGCCGCCCGGAACCTGAAGAAGGAGCGAACTCCCAGCTTCTCTGCCAGCGATGGTGACAGCGACGGGAGTGGCCCCACCTGTGGGCGGCGGCCAGGCTTGAAGCAGGAGGATGGTCCGCACATCCGTATCATGAAGAGAAGGTACTTGGACCAGGGCCGGACAGGAAGGCGCAAGGCTCAGATGGGGCTGGAGCTTCAGGCCTTCAGCTGCTCAGATGAGAGTGTCCACACCGGCCTCCCACACCTTCCCTCAGATGCTGGTCTTTTTGGGGTCCTGTGTGGGTCGCAGGCAGGAGCTGTTTCCTCATCTGCCCCCTGTCTGGCGTCCCCTCCCACCTCTGCTCTGCGGCGCTCACTGGCAGAGGCAGGTTGGCAGCAGTTGGGACCCAGAGGTCTGCACCTTCCTGGGCCGACGCTCCAGCTACCCTTGCTGACCGGGTCCCAGTCTGGCCAGAGAGCAGCTCTAGCAACAGGGAGCTCCATTCAGGCTCGTGACTGGCTGTGCAGAAGCAGCCTCGGCCCCCACCTGCGGTACAACAGGAGGGCTCCTCTGAGTGCACGGCAACAAGCAAGAGGGAGAAGGGGCCTCGGTCCTGTTCTTCCTGATGCGTGTCTGCTGAGGCCAGGAGCTGGCTTTGGCCCATGGGCCTGTCCTAGTGGGAGGCCCCAGCATGTTGAGCCAGTAGCAGGTGGTGCTGGGCATGGCAGCCGCCCTCGTTCACTGCCCAGGGCTGTGGCCCAGCGGGGCACTGACCCGAGACAGGTCTGCGCACGCCCTGCTATCCTGAGGCTGGGGTCAGGGGCCTCCAGAGCAACATGGACCTTCTGCTTCCCTTCCTGCAGAGTCCACACCCACTGGGACGTGAACATCTCTTTCCGAGAGGCGTCCTGCAGGTAGGAGCCGTGCTGTGCGTGCATAAGAGGGGGCCGTGACTCCCCTCCCTCCCTCCCACCCCTGACCGTGCCCTGCTGTCTGCTGTCCGCTGTCTCAGCGTGAGCTGATGCTGTGATGCTGGCTGAGTGTCTGCCAGGTTTGACATGTGCTGCAAGGTTGTCCCCCATCCCGGGAGGCAGACAGTGTTGCACCCAGTTGGGACTGAGGGACCCCAGACCCAGTCAGATGCAGCTCTCGGCAGCAGCTCAGGTGTGAGTTCTGGGCAGCCCGGCCCTGGAGTTAGAGTGCACTTCCTCCCATGTGAGACTGGCCATTTGAGCCCAAAAATGAGGCTGTCACCTCCCCCTTCCCACCCTCCTAGAGACCCACAAGGAGGTGAGAATGCTGATGTGTGAGTGGGGCCCTGAAGGGTGTGTAGGAGCTCTAAGGCGAGGGGATGTCTGCAGAGTAGAGGAACAGGGAAGGGCGTGTAGGAGGGACGAGGAGTGAACCTGGCAGCTCTGGTTCAGTTGGATGCTGAAGAGTCATGGATGCTGGGCCTGTGGGCACCGTCCTCCAGGCGGGAGCCACCGAAAGTTCTTGAGCAGGGCAGTGACCAGGTGTATGTTTGGAGAAGGTCCCTCTGGAGGCCTTCCTGGCAGACAGGGGATTGGATTCAGGCTGTGGAAGCAGGACGGTAGGGGGTGTGATTCCAGGATGTGGAAAGGAGATAAAAATGAAGAGCCCCGGGGAAGAGGTCAAGGGAGTTGGGGGACCCGAGTTCCTGGCTCCAGGGGGAAGCGAGTGGTAAGTCTGTGAACAGAGCCCAGCTGTGGATTCTGTCAATGGGGTCAGGTCTCACCCTGTGGCTTCCAGGGCAGCAAGGCAGGAAGGAGGCGTCTGCCACAAGGCCAGCTTCCTGGGGCCAGAGCCGTGAAGGCCCAGGGGACCTGCGTGTCTTGGCTCCACGCCAGATGTGTTATTATTTATGTCTCTGAGAATGTCTGGATCTCAGAGCCGAATTACAATAAAAACATCTTTAAACTTATTTCTACCTCATTTTGGGGTTGCCAGCTCACCTGATCATTTTTATGAACTGTCATGAACACTGATGACATTTTATGAGCCTTTTACATGGGACACTACAGAATACATTTGTCAGCGAGGCCTGTAGGGAAACCC',\n",
       " 'mutated_window': 'GGCCGAGGCGGGCGGATCACGAGGTCAGGAGATCGAGACCATCCTGACTAACACGGTGAAACCCGTCTCTACTAAAAATACAAAAAATTAGCCGGGCGTGGTGGCGGGTGCCTGTAGTCCCAGCTACTTGGGAGGCTGAGGCAGGAGAATGGCATGAACCCGGGAGGCGGAGCTTGCAGTGAGCCCAGATTGTGCCACCGCACTCCAGCCTGGGCAACAGAGTGAGACTCCGTCTCAAAAAACTAAAAAAGAAGAGAGGTGGGAGAGGAGAGGCTGTCAGAGCCTCTAAGCCCTGGTGCTTGGGCTGCAGAAGGGCAGAGCTAAGCGGGACTTCCCAGCACAGCACACTCCGGACAGGCTGTGGCTGTTGAAGGGACCCCCGAGCTCCAGCTGACACGCGGAGGCCCGGGCACAGACAGGCATCATACCTTCGGCCTTGGCCGCACTCTGTGGTCATTGGTGTTGGGGGCAGCCCAGGGTCAGGGCAGGGTCTCAGCCTCGGACCCCAGGCCCCACCCCTTGCCCAGCAGTGCTGCGTTTTCCCAGTGAGCTGTCGTGGAGAGAGCAGAGGGGACCCAGCGCAGGCCCAGTGGCCGGTGAGGGGAGACGTGGCTCTGGGACGGGGGCCTCCACCTGGGTGGGGGGATGCTCCAGCTTCCAGACCCTTGGGGAGGGGGCACTGCCCAAACTAAGCTGGCACTGGGGCTGTGCATTTGAAGGTGATGGTGGTTCTAGGTCTGAGGAGGACACCCTCCTAACAGCCTCATCCCCAAGCTCCGGGCTGTGTTGTGGCAATGGGAGGGAGGAAGTCTGAGGAGACCCTGGTGACTGAACGGAGGAGGGAGTGAGTTAGACGCTCTCAAGGGCTCTGCCACCTCCCGGAGCCAGCGGCCTGTTACTACATTTAAAAAAGCCTCCCGCCCACTGGAAAATAATCAATAACTTTCCTTTATCCCTGGGGGTGGCAGGACCTAGAAACACTGGAGGAGTCCGGAAGTGCCTGGGGCTGGGCCGGCGCTGGTGTGCTGTGCAGGGTGCCGCGGGCACGTCCGCCGCGTGTGTGCGTCAGCTCGGGGCTCGGCTGTGCTCTGCAGGGACCACAGCGGGCGTGTCTGTGCTCCCACCCGAGGCACCCACAGCTCCACACGCTCGTTCCGTGGGTGCAAAGGAGATGGGAGAAAGAAGCCCTGTGAGAAATGCGGGGCAGGGTTTGCGGAACAGGGGACCTGGGCTGGTGAGGGCTCCTCGTCTGGTGACCTGTGAGCCCCGGGGCCTGCAGTCTGCGAGGGTTCAGCTCAGACAGTTGCCAGTGGCCTTGCACCAGGCTGCAGCTGCCCCTGAGCCGGGCTGTGCGTGGCGCTGATGAAATAGAAAAGGGCATTCGCTTGTCAACGTTGGCATCGGTGGCAGGGTGTGGTGGGCAGAAGGGTCACAAAGTACGGGTGGGATTGGCAGGCAGATACACGGAGGGAACGTGCGCATTTGAGTGCACGTCCACCAGCACCAGCCCCAGGCCACAGGCAGATCCCAGGAGACACGCAGGGGCCCTAAGAAGGGAGCTGGGAATGAGGGGCCACACAAGCCCGGGACGGAGGCCTGTCGCACATGGGGTGGCCCCGACTCAGGCCCTGGAGTTGGCCAGGACCCTCTAGCATCCTCAAGGGCTGGGCCAACCAGGCTGGCGTGGGGTGGGGCAGGGGAGGGCTGAGCCAGTGGGCGTCGTCTGTAGGGGGATGCCCAACTGCGGCCCCGTCTCTCGGCTCTCCTCTGGGTCTCTGGCCAGCTGTGGCTCCTGCTGGCCCCAGGCGCATCCCAGAGGCAGGTAGAGGGAGGATGGCTGCTCTGAGGGCACCTCTGCCGTGCTTGGGGCTCGGCCTGGGGTGCGAGACCAGGGCAGACCCCCGGGAGATGGAACGGCCCGGTCCAGCCCCACCTTCCTCTCCTCCTGCCCCACCTTCCTCTCCTCCTGCCCCACCTTCCTCTCCTCCTGCCCCACCAGAACCGGGGGCGGCTGGCAGACAAGAGGACAGTCGCCCTGCCTGCCGCCCAGAACCTGAAGAAGGAGCGAACTCCCAGCTTCTCTGCCAGCGATGGTGACAGCGACGGGAGTGGCCCCACCTGTGGGCGGCGGCCAGGCTTGAAGCAGGAGGATGGTCCGCACATCCGTATCATGAAGAGAAGGTACTTGGACCAGGGCCGGACAGGAAGGCGCAAGGCTCAGATGGGGCTGGAGCTTCAGGCCTTCAGCTGCTCAGATGAGAGTGTCCACACCGGCCTCCCACACCTTCCCTCAGATGCTGGTCTTTTTGGGGTCCTGTGTGGGTCGCAGGCAGGAGCTGTTTCCTCATCTGCCCCCTGTCTGGCGTCCCCTCCCACCTCTGCTCTGCGGCGCTCACTGGCAGAGGCAGGTTGGCAGCAGTTGGGACCCAGAGGTCTGCACCTTCCTGGGCCGACGCTCCAGCTACCCTTGCTGACCGGGTCCCAGTCTGGCCAGAGAGCAGCTCTAGCAACAGGGAGCTCCATTCAGGCTCGTGACTGGCTGTGCAGAAGCAGCCTCGGCCCCCACCTGCGGTACAACAGGAGGGCTCCTCTGAGTGCACGGCAACAAGCAAGAGGGAGAAGGGGCCTCGGTCCTGTTCTTCCTGATGCGTGTCTGCTGAGGCCAGGAGCTGGCTTTGGCCCATGGGCCTGTCCTAGTGGGAGGCCCCAGCATGTTGAGCCAGTAGCAGGTGGTGCTGGGCATGGCAGCCGCCCTCGTTCACTGCCCAGGGCTGTGGCCCAGCGGGGCACTGACCCGAGACAGGTCTGCGCACGCCCTGCTATCCTGAGGCTGGGGTCAGGGGCCTCCAGAGCAACATGGACCTTCTGCTTCCCTTCCTGCAGAGTCCACACCCACTGGGACGTGAACATCTCTTTCCGAGAGGCGTCCTGCAGGTAGGAGCCGTGCTGTGCGTGCATAAGAGGGGGCCGTGACTCCCCTCCCTCCCTCCCACCCCTGACCGTGCCCTGCTGTCTGCTGTCCGCTGTCTCAGCGTGAGCTGATGCTGTGATGCTGGCTGAGTGTCTGCCAGGTTTGACATGTGCTGCAAGGTTGTCCCCCATCCCGGGAGGCAGACAGTGTTGCACCCAGTTGGGACTGAGGGACCCCAGACCCAGTCAGATGCAGCTCTCGGCAGCAGCTCAGGTGTGAGTTCTGGGCAGCCCGGCCCTGGAGTTAGAGTGCACTTCCTCCCATGTGAGACTGGCCATTTGAGCCCAAAAATGAGGCTGTCACCTCCCCCTTCCCACCCTCCTAGAGACCCACAAGGAGGTGAGAATGCTGATGTGTGAGTGGGGCCCTGAAGGGTGTGTAGGAGCTCTAAGGCGAGGGGATGTCTGCAGAGTAGAGGAACAGGGAAGGGCGTGTAGGAGGGACGAGGAGTGAACCTGGCAGCTCTGGTTCAGTTGGATGCTGAAGAGTCATGGATGCTGGGCCTGTGGGCACCGTCCTCCAGGCGGGAGCCACCGAAAGTTCTTGAGCAGGGCAGTGACCAGGTGTATGTTTGGAGAAGGTCCCTCTGGAGGCCTTCCTGGCAGACAGGGGATTGGATTCAGGCTGTGGAAGCAGGACGGTAGGGGGTGTGATTCCAGGATGTGGAAAGGAGATAAAAATGAAGAGCCCCGGGGAAGAGGTCAAGGGAGTTGGGGGACCCGAGTTCCTGGCTCCAGGGGGAAGCGAGTGGTAAGTCTGTGAACAGAGCCCAGCTGTGGATTCTGTCAATGGGGTCAGGTCTCACCCTGTGGCTTCCAGGGCAGCAAGGCAGGAAGGAGGCGTCTGCCACAAGGCCAGCTTCCTGGGGCCAGAGCCGTGAAGGCCCAGGGGACCTGCGTGTCTTGGCTCCACGCCAGATGTGTTATTATTTATGTCTCTGAGAATGTCTGGATCTCAGAGCCGAATTACAATAAAAACATCTTTAAACTTATTTCTACCTCATTTTGGGGTTGCCAGCTCACCTGATCATTTTTATGAACTGTCATGAACACTGATGACATTTTATGAGCCTTTTACATGGGACACTACAGAATACATTTGTCAGCGAGGCCTGTAGGGAAACCC',\n",
       " 'cleaned_pathogenicity': 'benign',\n",
       " 'disease_name': 'SAMD11-related_disorder|not_provided',\n",
       " 'gene_name': 'SAMD11',\n",
       " 'gene_desc': 'sterile alpha motif domain containing 11',\n",
       " 'chromosome': '1',\n",
       " 'chromosome_position': '930204',\n",
       " 'variant_type': 'SNV',\n",
       " 'clinvar_link': 'https://www.ncbi.nlm.nih.gov/clinvar/variation/1170208/',\n",
       " 'gene_id': '148398',\n",
       " 'mutation_instruction': 'G>A',\n",
       " 'pathogenicity': 'benign',\n",
       " 'review_status': 'criteria_provided,_multiple_submitters,_no_conflicts'}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[0].to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "answer\n",
       "benign                                                                                                                                                                                                         80\n",
       "pathogenic; ['Intellectual_disability,_X-linked_102']                                                                                                                                                           1\n",
       "pathogenic; ['Familial_adenomatous_polyposis_2', 'Hereditary_cancer-predisposing_syndrome']                                                                                                                     1\n",
       "pathogenic; ['Familial_thoracic_aortic_aneurysm_and_aortic_dissection', 'Hereditary_cancer-predisposing_syndrome', 'Juvenile_polyposis_syndrome']                                                               1\n",
       "pathogenic; ['Familial_cancer_of_breast', 'Hereditary_cancer-predisposing_syndrome']                                                                                                                            1\n",
       "pathogenic; ['Bardet-Biedl_syndrome_2', 'Retinitis_pigmentosa_74']                                                                                                                                              1\n",
       "pathogenic; ['Early-onset_retinal_dystrophy', 'Leber_congenital_amaurosis', 'Leber_congenital_amaurosis_8', 'Pigmented_paravenous_retinochoroidal_atrophy', 'Retinal_dystrophy', 'Retinitis_pigmentosa_12']     1\n",
       "pathogenic; ['Autosomal_recessive_limb-girdle_muscular_dystrophy_type_2E']                                                                                                                                      1\n",
       "pathogenic; ['Childhood_Onset_Dystonias', 'Dystonia,_childhood-onset,_with_optic_atrophy_and_basal_ganglia_abnormalities', 'MECR-related_disorder', 'Mitochondrial_disease', 'Optic_atrophy']                   1\n",
       "pathogenic; ['Autoimmune_thyroid_disease,_susceptibility_to,_3', 'Iodotyrosyl_coupling_defect']                                                                                                                 1\n",
       "pathogenic; ['Duchenne_muscular_dystrophy']                                                                                                                                                                     1\n",
       "pathogenic; ['Ataxia-telangiectasia_syndrome', 'Hereditary_cancer-predisposing_syndrome']                                                                                                                       1\n",
       "pathogenic; ['Autosomal_dominant_nonsyndromic_hearing_loss_6', 'Cataract_41', 'Type_2_diabetes_mellitus', 'Wolfram-like_syndrome', 'Wolfram_syndrome_1']                                                        1\n",
       "pathogenic; ['Autosomal_recessive_limb-girdle_muscular_dystrophy_type_2B', 'Distal_myopathy_with_anterior_tibial_onset', 'Miyoshi_muscular_dystrophy_1']                                                        1\n",
       "pathogenic; ['Breast-ovarian_cancer,_familial,_susceptibility_to,_1']                                                                                                                                           1\n",
       "pathogenic; ['Arterial_calcification,_generalized,_of_infancy,_2', 'Autosomal_recessive_inherited_pseudoxanthoma_elasticum', 'Pseudoxanthoma_elasticum,_forme_fruste']                                          1\n",
       "pathogenic; ['Hereditary_cancer-predisposing_syndrome', 'Juvenile_polyposis_syndrome']                                                                                                                          1\n",
       "pathogenic; ['Monogenic_diabetes']                                                                                                                                                                              1\n",
       "pathogenic; ['Autosomal_recessive_osteopetrosis_1']                                                                                                                                                             1\n",
       "pathogenic; ['Autosomal_dominant_nonsyndromic_hearing_loss_11', 'Autosomal_recessive_nonsyndromic_hearing_loss_2', 'Rare_genetic_deafness', 'Retinal_dystrophy', 'Usher_syndrome_type_1']                       1\n",
       "pathogenic; ['Wilson_disease']                                                                                                                                                                                  1\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_df['answer'].sample(100).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "visualization of table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clinvar_id</th>\n",
       "      <th>original_window</th>\n",
       "      <th>mutated_window</th>\n",
       "      <th>cleaned_pathogenicity</th>\n",
       "      <th>disease_name</th>\n",
       "      <th>variant_type</th>\n",
       "      <th>clinvar_link</th>\n",
       "      <th>mutation_instruction</th>\n",
       "      <th>pathogenicity</th>\n",
       "      <th>review_status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1170208</td>\n",
       "      <td>GGCCGAGGCGGGCGGATCACGAGGTCAGGAGATCGAGACCATCCTG...</td>\n",
       "      <td>GGCCGAGGCGGGCGGATCACGAGGTCAGGAGATCGAGACCATCCTG...</td>\n",
       "      <td>benign</td>\n",
       "      <td>SAMD11-related_disorder|not_provided</td>\n",
       "      <td>SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>G&gt;A</td>\n",
       "      <td>benign</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1170010</td>\n",
       "      <td>CCTGTAGTCCCAGCTACTTGGGAGGCTGAGGCAGGAGAATGGCATG...</td>\n",
       "      <td>CCTGTAGTCCCAGCTACTTGGGAGGCTGAGGCAGGAGAATGGCATG...</td>\n",
       "      <td>benign</td>\n",
       "      <td>SAMD11-related_disorder|not_provided</td>\n",
       "      <td>SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>C&gt;T</td>\n",
       "      <td>benign</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1170044</td>\n",
       "      <td>GAGGGAGTGAGTTAGACGCTCTCAAGGGCTCTGCCACCTCCCGGAG...</td>\n",
       "      <td>GAGGGAGTGAGTTAGACGCTCTCAAGGGCTCTGCCACCTCCCGGAG...</td>\n",
       "      <td>benign</td>\n",
       "      <td>not_provided|SAMD11-related_disorder</td>\n",
       "      <td>SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>C&gt;T</td>\n",
       "      <td>benign</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1170011</td>\n",
       "      <td>AGCCGTCATCTAGGTCTCCTGGAAGGTTTAGAGCCCAGCCTGGGAG...</td>\n",
       "      <td>AGCCGTCATCTAGGTCTCCTGGAAGGTTTAGAGCCCAGCCTGGGAG...</td>\n",
       "      <td>benign</td>\n",
       "      <td>SAMD11-related_disorder|not_provided</td>\n",
       "      <td>SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>C&gt;G</td>\n",
       "      <td>benign</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1169668</td>\n",
       "      <td>GGTTTAGAGCCCAGCCTGGGAGTCTTTGGTGCTGAAACGGATCTGC...</td>\n",
       "      <td>GGTTTAGAGCCCAGCCTGGGAGTCTTTGGTGCTGAAACGGATCTGC...</td>\n",
       "      <td>benign</td>\n",
       "      <td>not_provided</td>\n",
       "      <td>SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>C&gt;T</td>\n",
       "      <td>benign</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342875</th>\n",
       "      <td>522717</td>\n",
       "      <td>TGTCATCCCTCTTATTAATCATCATCCTAGCCCTAAGTCTGGCCTA...</td>\n",
       "      <td>TGTCATCCCTCTTATTAATCATCATCCTAGCCCTAAGTCTGGCCTA...</td>\n",
       "      <td>benign</td>\n",
       "      <td>Mitochondrial_disease|not_specified</td>\n",
       "      <td>SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>G&gt;A</td>\n",
       "      <td>benign</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342878</th>\n",
       "      <td>65510</td>\n",
       "      <td>CTAAAACTAATCGTCCCAACAATTATATTACTACCACTGACATGAC...</td>\n",
       "      <td>CTAAAACTAATCGTCCCAACAATTATATTACTACCACTGACATGAC...</td>\n",
       "      <td>benign</td>\n",
       "      <td>Leber_optic_atrophy|Leigh_syndrome|Mitochondri...</td>\n",
       "      <td>SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>T&gt;C</td>\n",
       "      <td>benign</td>\n",
       "      <td>reviewed_by_expert_panel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342905</th>\n",
       "      <td>140592</td>\n",
       "      <td>AGTTACAATCGGCATCAACCAACCACACCTAGCATTCCTGCACATC...</td>\n",
       "      <td>AGTTACAATCGGCATCAACCAACCACACCTAGCATTCCTGCACATC...</td>\n",
       "      <td>benign</td>\n",
       "      <td>Familial_cancer_of_breast|Mitochondrial_diseas...</td>\n",
       "      <td>SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>A&gt;G</td>\n",
       "      <td>benign</td>\n",
       "      <td>reviewed_by_expert_panel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342907</th>\n",
       "      <td>235623</td>\n",
       "      <td>TAAACGCCTGGCAGCCGGAAGCCTATTCGCAGGATTTCTCATTACT...</td>\n",
       "      <td>TAAACGCCTGGCAGCCGGAAGCCTATTCGCAGGATTTCTCATTACT...</td>\n",
       "      <td>benign</td>\n",
       "      <td>Leigh_syndrome|not_provided</td>\n",
       "      <td>SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>A&gt;G</td>\n",
       "      <td>benign</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342909</th>\n",
       "      <td>252455</td>\n",
       "      <td>AGCCCTAGACCTCAACTACCTAACCAACAAACTTAAAATAAAATCC...</td>\n",
       "      <td>AGCCCTAGACCTCAACTACCTAACCAACAAACTTAAAATAAAATCC...</td>\n",
       "      <td>benign</td>\n",
       "      <td>not_specified|Leigh_syndrome</td>\n",
       "      <td>SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>G&gt;C</td>\n",
       "      <td>benign</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>93800 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       clinvar_id                                    original_window  \\\n",
       "0         1170208  GGCCGAGGCGGGCGGATCACGAGGTCAGGAGATCGAGACCATCCTG...   \n",
       "2         1170010  CCTGTAGTCCCAGCTACTTGGGAGGCTGAGGCAGGAGAATGGCATG...   \n",
       "3         1170044  GAGGGAGTGAGTTAGACGCTCTCAAGGGCTCTGCCACCTCCCGGAG...   \n",
       "5         1170011  AGCCGTCATCTAGGTCTCCTGGAAGGTTTAGAGCCCAGCCTGGGAG...   \n",
       "7         1169668  GGTTTAGAGCCCAGCCTGGGAGTCTTTGGTGCTGAAACGGATCTGC...   \n",
       "...           ...                                                ...   \n",
       "342875     522717  TGTCATCCCTCTTATTAATCATCATCCTAGCCCTAAGTCTGGCCTA...   \n",
       "342878      65510  CTAAAACTAATCGTCCCAACAATTATATTACTACCACTGACATGAC...   \n",
       "342905     140592  AGTTACAATCGGCATCAACCAACCACACCTAGCATTCCTGCACATC...   \n",
       "342907     235623  TAAACGCCTGGCAGCCGGAAGCCTATTCGCAGGATTTCTCATTACT...   \n",
       "342909     252455  AGCCCTAGACCTCAACTACCTAACCAACAAACTTAAAATAAAATCC...   \n",
       "\n",
       "                                           mutated_window  \\\n",
       "0       GGCCGAGGCGGGCGGATCACGAGGTCAGGAGATCGAGACCATCCTG...   \n",
       "2       CCTGTAGTCCCAGCTACTTGGGAGGCTGAGGCAGGAGAATGGCATG...   \n",
       "3       GAGGGAGTGAGTTAGACGCTCTCAAGGGCTCTGCCACCTCCCGGAG...   \n",
       "5       AGCCGTCATCTAGGTCTCCTGGAAGGTTTAGAGCCCAGCCTGGGAG...   \n",
       "7       GGTTTAGAGCCCAGCCTGGGAGTCTTTGGTGCTGAAACGGATCTGC...   \n",
       "...                                                   ...   \n",
       "342875  TGTCATCCCTCTTATTAATCATCATCCTAGCCCTAAGTCTGGCCTA...   \n",
       "342878  CTAAAACTAATCGTCCCAACAATTATATTACTACCACTGACATGAC...   \n",
       "342905  AGTTACAATCGGCATCAACCAACCACACCTAGCATTCCTGCACATC...   \n",
       "342907  TAAACGCCTGGCAGCCGGAAGCCTATTCGCAGGATTTCTCATTACT...   \n",
       "342909  AGCCCTAGACCTCAACTACCTAACCAACAAACTTAAAATAAAATCC...   \n",
       "\n",
       "       cleaned_pathogenicity  \\\n",
       "0                     benign   \n",
       "2                     benign   \n",
       "3                     benign   \n",
       "5                     benign   \n",
       "7                     benign   \n",
       "...                      ...   \n",
       "342875                benign   \n",
       "342878                benign   \n",
       "342905                benign   \n",
       "342907                benign   \n",
       "342909                benign   \n",
       "\n",
       "                                             disease_name variant_type  \\\n",
       "0                    SAMD11-related_disorder|not_provided          SNV   \n",
       "2                    SAMD11-related_disorder|not_provided          SNV   \n",
       "3                    not_provided|SAMD11-related_disorder          SNV   \n",
       "5                    SAMD11-related_disorder|not_provided          SNV   \n",
       "7                                            not_provided          SNV   \n",
       "...                                                   ...          ...   \n",
       "342875                Mitochondrial_disease|not_specified          SNV   \n",
       "342878  Leber_optic_atrophy|Leigh_syndrome|Mitochondri...          SNV   \n",
       "342905  Familial_cancer_of_breast|Mitochondrial_diseas...          SNV   \n",
       "342907                        Leigh_syndrome|not_provided          SNV   \n",
       "342909                       not_specified|Leigh_syndrome          SNV   \n",
       "\n",
       "                                             clinvar_link  \\\n",
       "0       https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "2       https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "3       https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "5       https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "7       https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "...                                                   ...   \n",
       "342875  https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "342878  https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "342905  https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "342907  https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "342909  https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "\n",
       "       mutation_instruction pathogenicity  \\\n",
       "0                       G>A        benign   \n",
       "2                       C>T        benign   \n",
       "3                       C>T        benign   \n",
       "5                       C>G        benign   \n",
       "7                       C>T        benign   \n",
       "...                     ...           ...   \n",
       "342875                  G>A        benign   \n",
       "342878                  T>C        benign   \n",
       "342905                  A>G        benign   \n",
       "342907                  A>G        benign   \n",
       "342909                  G>C        benign   \n",
       "\n",
       "                                            review_status  \n",
       "0       criteria_provided,_multiple_submitters,_no_con...  \n",
       "2       criteria_provided,_multiple_submitters,_no_con...  \n",
       "3       criteria_provided,_multiple_submitters,_no_con...  \n",
       "5       criteria_provided,_multiple_submitters,_no_con...  \n",
       "7       criteria_provided,_multiple_submitters,_no_con...  \n",
       "...                                                   ...  \n",
       "342875  criteria_provided,_multiple_submitters,_no_con...  \n",
       "342878                           reviewed_by_expert_panel  \n",
       "342905                           reviewed_by_expert_panel  \n",
       "342907  criteria_provided,_multiple_submitters,_no_con...  \n",
       "342909  criteria_provided,_multiple_submitters,_no_con...  \n",
       "\n",
       "[93800 rows x 10 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df['pathogenicity']=='benign']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clinvar_id</th>\n",
       "      <th>original_window</th>\n",
       "      <th>mutated_window</th>\n",
       "      <th>cleaned_pathogenicity</th>\n",
       "      <th>disease_name</th>\n",
       "      <th>variant_type</th>\n",
       "      <th>clinvar_link</th>\n",
       "      <th>mutation_instruction</th>\n",
       "      <th>pathogenicity</th>\n",
       "      <th>review_status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>1185392</td>\n",
       "      <td>TTATTGATGTGAAATTCATATAACATAAAACTAACCATTTTAAAGA...</td>\n",
       "      <td>TTATTGATGTGAAATTCATATAACATAAAACTAACCATTTTAAAGA...</td>\n",
       "      <td>benign</td>\n",
       "      <td>Mendelian_susceptibility_to_mycobacterial_dise...</td>\n",
       "      <td>non_SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>T&gt;TA</td>\n",
       "      <td>benign</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>666960</td>\n",
       "      <td>TGGTGCAGGGAGGTGACTGGGTCCTTGGCCATGGGGTTGGGACCTG...</td>\n",
       "      <td>TGGTGCAGGGAGGTGACTGGGTCCTTGGCCATGGGGTTGGGACCTG...</td>\n",
       "      <td>pathogenic</td>\n",
       "      <td>Congenital_myasthenic_syndrome|Congenital_myas...</td>\n",
       "      <td>non_SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>G&gt;GGGGCC</td>\n",
       "      <td>pathogenic/likely_pathogenic</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>970311</td>\n",
       "      <td>ATCAGCAGGTGCCCGTTGGATTTGGACTGGGAGTCCCAGGGCCTTG...</td>\n",
       "      <td>ATCAGCAGGTGCCCGTTGGATTTGGACTGGGAGTCCCAGGGCCTTG...</td>\n",
       "      <td>pathogenic</td>\n",
       "      <td>Congenital_myasthenic_syndrome_8</td>\n",
       "      <td>non_SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>G&gt;GC</td>\n",
       "      <td>pathogenic/likely_pathogenic</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>930633</td>\n",
       "      <td>GTGCCTGAGGCAGCTTTGTTGGCCACGTTGAGGTCTGGTGATGGGA...</td>\n",
       "      <td>GTGCCTGAGGCAGCTTTGTTGGCCACGTTGAGGTCTGGTGATGGGA...</td>\n",
       "      <td>pathogenic</td>\n",
       "      <td>Presynaptic_congenital_myasthenic_syndrome|Con...</td>\n",
       "      <td>non_SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>CGCTCCGGCCAGTGCCAGGGTCGAGGTGAGCGGCTCCCCCGGGGGA...</td>\n",
       "      <td>likely_pathogenic</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>263160</td>\n",
       "      <td>TCGCGGGACCCCTGCTCCAACGTGACCTGCAGCTTCGGCAGCACCT...</td>\n",
       "      <td>TCGCGGGACCCCTGCTCCAACGTGACCTGCAGCTTCGGCAGCACCT...</td>\n",
       "      <td>benign</td>\n",
       "      <td>not_provided|not_specified|Congenital_myasthen...</td>\n",
       "      <td>non_SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>CCT&gt;C</td>\n",
       "      <td>benign</td>\n",
       "      <td>criteria_provided,_multiple_submitters,_no_con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342844</th>\n",
       "      <td>9654</td>\n",
       "      <td>TACATAAAATCTAGACAAAAAAGGAAGGAATCGAACCCCCCAAAGC...</td>\n",
       "      <td>TACATAAAATCTAGACAAAAAAGGAAGGAATCGAACCCCCCAAAGC...</td>\n",
       "      <td>pathogenic</td>\n",
       "      <td>Mitochondrial_disease|Mitochondrial_complex_IV...</td>\n",
       "      <td>non_SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>TTTTTTCTTCGCAGGA&gt;T</td>\n",
       "      <td>likely_pathogenic</td>\n",
       "      <td>reviewed_by_expert_panel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342845</th>\n",
       "      <td>9656</td>\n",
       "      <td>CAAGCCAACCCCATGGCCTCCATGACTTTTTCAAAAAGGTATTAGA...</td>\n",
       "      <td>CAAGCCAACCCCATGGCCTCCATGACTTTTTCAAAAAGGTATTAGA...</td>\n",
       "      <td>pathogenic</td>\n",
       "      <td>Mitochondrial_disease|Mitochondrial_complex_IV...</td>\n",
       "      <td>non_SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>A&gt;AC</td>\n",
       "      <td>likely_pathogenic</td>\n",
       "      <td>reviewed_by_expert_panel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342876</th>\n",
       "      <td>693440</td>\n",
       "      <td>ATGAGTGACTACAAAAAGGATTAGACTGAACCGAATTGGTATATAG...</td>\n",
       "      <td>ATGAGTGACTACAAAAAGGATTAGACTGAACCGAATTGGTATATAG...</td>\n",
       "      <td>pathogenic</td>\n",
       "      <td>Mitochondrial_myopathy_with_reversible_cytochr...</td>\n",
       "      <td>non_SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>CA&gt;C</td>\n",
       "      <td>likely_pathogenic</td>\n",
       "      <td>reviewed_by_expert_panel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342895</th>\n",
       "      <td>800503</td>\n",
       "      <td>ACCTTTATTATCAGTCTCTTCCCCACAACAATATTCATGTGCCTAG...</td>\n",
       "      <td>ACCTTTATTATCAGTCTCTTCCCCACAACAATATTCATGTGCCTAG...</td>\n",
       "      <td>pathogenic</td>\n",
       "      <td>Mitochondrial_disease</td>\n",
       "      <td>non_SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>CTA&gt;C</td>\n",
       "      <td>likely_pathogenic</td>\n",
       "      <td>reviewed_by_expert_panel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342901</th>\n",
       "      <td>9686</td>\n",
       "      <td>TACCGCTAACAACCTATTCCAACTGTTCATCGGCTGAGAGGGCGTA...</td>\n",
       "      <td>TACCGCTAACAACCTATTCCAACTGTTCATCGGCTGAGAGGGCGTA...</td>\n",
       "      <td>pathogenic</td>\n",
       "      <td>Mitochondrial_disease|Parkinsonism/MELAS_overl...</td>\n",
       "      <td>non_SNV</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/clinvar/variation...</td>\n",
       "      <td>AAATT&gt;A</td>\n",
       "      <td>likely_pathogenic</td>\n",
       "      <td>reviewed_by_expert_panel</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>36097 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       clinvar_id                                    original_window  \\\n",
       "42        1185392  TTATTGATGTGAAATTCATATAACATAAAACTAACCATTTTAAAGA...   \n",
       "67         666960  TGGTGCAGGGAGGTGACTGGGTCCTTGGCCATGGGGTTGGGACCTG...   \n",
       "69         970311  ATCAGCAGGTGCCCGTTGGATTTGGACTGGGAGTCCCAGGGCCTTG...   \n",
       "80         930633  GTGCCTGAGGCAGCTTTGTTGGCCACGTTGAGGTCTGGTGATGGGA...   \n",
       "90         263160  TCGCGGGACCCCTGCTCCAACGTGACCTGCAGCTTCGGCAGCACCT...   \n",
       "...           ...                                                ...   \n",
       "342844       9654  TACATAAAATCTAGACAAAAAAGGAAGGAATCGAACCCCCCAAAGC...   \n",
       "342845       9656  CAAGCCAACCCCATGGCCTCCATGACTTTTTCAAAAAGGTATTAGA...   \n",
       "342876     693440  ATGAGTGACTACAAAAAGGATTAGACTGAACCGAATTGGTATATAG...   \n",
       "342895     800503  ACCTTTATTATCAGTCTCTTCCCCACAACAATATTCATGTGCCTAG...   \n",
       "342901       9686  TACCGCTAACAACCTATTCCAACTGTTCATCGGCTGAGAGGGCGTA...   \n",
       "\n",
       "                                           mutated_window  \\\n",
       "42      TTATTGATGTGAAATTCATATAACATAAAACTAACCATTTTAAAGA...   \n",
       "67      TGGTGCAGGGAGGTGACTGGGTCCTTGGCCATGGGGTTGGGACCTG...   \n",
       "69      ATCAGCAGGTGCCCGTTGGATTTGGACTGGGAGTCCCAGGGCCTTG...   \n",
       "80      GTGCCTGAGGCAGCTTTGTTGGCCACGTTGAGGTCTGGTGATGGGA...   \n",
       "90      TCGCGGGACCCCTGCTCCAACGTGACCTGCAGCTTCGGCAGCACCT...   \n",
       "...                                                   ...   \n",
       "342844  TACATAAAATCTAGACAAAAAAGGAAGGAATCGAACCCCCCAAAGC...   \n",
       "342845  CAAGCCAACCCCATGGCCTCCATGACTTTTTCAAAAAGGTATTAGA...   \n",
       "342876  ATGAGTGACTACAAAAAGGATTAGACTGAACCGAATTGGTATATAG...   \n",
       "342895  ACCTTTATTATCAGTCTCTTCCCCACAACAATATTCATGTGCCTAG...   \n",
       "342901  TACCGCTAACAACCTATTCCAACTGTTCATCGGCTGAGAGGGCGTA...   \n",
       "\n",
       "       cleaned_pathogenicity  \\\n",
       "42                    benign   \n",
       "67                pathogenic   \n",
       "69                pathogenic   \n",
       "80                pathogenic   \n",
       "90                    benign   \n",
       "...                      ...   \n",
       "342844            pathogenic   \n",
       "342845            pathogenic   \n",
       "342876            pathogenic   \n",
       "342895            pathogenic   \n",
       "342901            pathogenic   \n",
       "\n",
       "                                             disease_name variant_type  \\\n",
       "42      Mendelian_susceptibility_to_mycobacterial_dise...      non_SNV   \n",
       "67      Congenital_myasthenic_syndrome|Congenital_myas...      non_SNV   \n",
       "69                       Congenital_myasthenic_syndrome_8      non_SNV   \n",
       "80      Presynaptic_congenital_myasthenic_syndrome|Con...      non_SNV   \n",
       "90      not_provided|not_specified|Congenital_myasthen...      non_SNV   \n",
       "...                                                   ...          ...   \n",
       "342844  Mitochondrial_disease|Mitochondrial_complex_IV...      non_SNV   \n",
       "342845  Mitochondrial_disease|Mitochondrial_complex_IV...      non_SNV   \n",
       "342876  Mitochondrial_myopathy_with_reversible_cytochr...      non_SNV   \n",
       "342895                              Mitochondrial_disease      non_SNV   \n",
       "342901  Mitochondrial_disease|Parkinsonism/MELAS_overl...      non_SNV   \n",
       "\n",
       "                                             clinvar_link  \\\n",
       "42      https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "67      https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "69      https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "80      https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "90      https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "...                                                   ...   \n",
       "342844  https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "342845  https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "342876  https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "342895  https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "342901  https://www.ncbi.nlm.nih.gov/clinvar/variation...   \n",
       "\n",
       "                                     mutation_instruction  \\\n",
       "42                                                   T>TA   \n",
       "67                                               G>GGGGCC   \n",
       "69                                                   G>GC   \n",
       "80      CGCTCCGGCCAGTGCCAGGGTCGAGGTGAGCGGCTCCCCCGGGGGA...   \n",
       "90                                                  CCT>C   \n",
       "...                                                   ...   \n",
       "342844                                 TTTTTTCTTCGCAGGA>T   \n",
       "342845                                               A>AC   \n",
       "342876                                               CA>C   \n",
       "342895                                              CTA>C   \n",
       "342901                                            AAATT>A   \n",
       "\n",
       "                       pathogenicity  \\\n",
       "42                            benign   \n",
       "67      pathogenic/likely_pathogenic   \n",
       "69      pathogenic/likely_pathogenic   \n",
       "80                 likely_pathogenic   \n",
       "90                            benign   \n",
       "...                              ...   \n",
       "342844             likely_pathogenic   \n",
       "342845             likely_pathogenic   \n",
       "342876             likely_pathogenic   \n",
       "342895             likely_pathogenic   \n",
       "342901             likely_pathogenic   \n",
       "\n",
       "                                            review_status  \n",
       "42      criteria_provided,_multiple_submitters,_no_con...  \n",
       "67      criteria_provided,_multiple_submitters,_no_con...  \n",
       "69      criteria_provided,_multiple_submitters,_no_con...  \n",
       "80      criteria_provided,_multiple_submitters,_no_con...  \n",
       "90      criteria_provided,_multiple_submitters,_no_con...  \n",
       "...                                                   ...  \n",
       "342844                           reviewed_by_expert_panel  \n",
       "342845                           reviewed_by_expert_panel  \n",
       "342876                           reviewed_by_expert_panel  \n",
       "342895                           reviewed_by_expert_panel  \n",
       "342901                           reviewed_by_expert_panel  \n",
       "\n",
       "[36097 rows x 10 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df['variant_type']=='non_SNV']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "variant_type\n",
       "SNV        306816\n",
       "non_SNV     36097\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['variant_type'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['likely_benign', 'benign', 'benign/likely_benign', 'pathogenic',\n",
       "       'pathogenic/likely_pathogenic', 'likely_pathogenic',\n",
       "       'pathogenic|drug_response', 'likely_pathogenic|drug_response',\n",
       "       'benign/likely_benign|other', 'likely_benign|other', 'benign|other',\n",
       "       'pathogenic/likely_pathogenic|other', 'pathogenic|other',\n",
       "       'benign|association', 'likely_benign|drug_response|other',\n",
       "       'pathogenic/likely_pathogenic|risk_factor', 'benign|drug_response',\n",
       "       'benign/likely_benign|drug_response|other',\n",
       "       'likely_pathogenic|risk_factor', 'pathogenic|risk_factor',\n",
       "       'benign/likely_benign|drug_response', 'benign|risk_factor',\n",
       "       'likely_benign|association', 'benign/likely_benign|other|risk_factor',\n",
       "       'benign/likely_benign|association', 'likely_pathogenic|affects',\n",
       "       'likely_pathogenic|other', 'benign/likely_benign|risk_factor',\n",
       "       'likely_pathogenic|association',\n",
       "       'pathogenic/likely_pathogenic|association',\n",
       "       'benign|confers_sensitivity', 'likely_benign|risk_factor'],\n",
       "      dtype='object', name='pathogenicity')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['pathogenicity'].value_counts().keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total variants: 3,493,400\n",
      "\n",
      "Variant type counts:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "variant_type\n",
       "SNV        3226063\n",
       "non_SNV     267337\n",
       "Name: count, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Pathogenicity counts:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "pathogenicity\n",
       "not_pathogenic    3043681\n",
       "pathogenic         449719\n",
       "Name: count, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Top 10 disease names:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "disease_name\n",
       "not_provided                               861927\n",
       "not_specified                              719547\n",
       "Inborn_genetic_diseases                    133139\n",
       "Hereditary_cancer-predisposing_syndrome     47592\n",
       "Cardiovascular_phenotype                    25149\n",
       "Primary_ciliary_dyskinesia                  17996\n",
       "Inborn_genetic_diseases|not_provided        16863\n",
       "not_specified|not_provided                  16518\n",
       "not_provided|Inborn_genetic_diseases        15874\n",
       "not_provided|not_specified                  14489\n",
       "Name: count, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Length‐difference (alt − ref) distribution:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "len_diff\n",
       "-2046    1\n",
       "-2037    1\n",
       "-2032    1\n",
       "-2031    1\n",
       "-2030    1\n",
       "        ..\n",
       " 1951    1\n",
       " 1989    1\n",
       " 1992    1\n",
       " 2004    1\n",
       " 2019    1\n",
       "Name: count, Length: 1266, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "SNVs: 3,226,063  →  Transitions: 2,104,260   Transversions: 1,121,803\n",
      "\n",
      "Original‐window GC content (sample):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "count    10000.000000\n",
       "mean         0.471380\n",
       "std          0.094873\n",
       "min          0.244629\n",
       "25%          0.389404\n",
       "50%          0.461914\n",
       "75%          0.548340\n",
       "max          0.744385\n",
       "Name: orig_gc, dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Mutated‐window GC content (sample):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "count    10000.000000\n",
       "mean         0.471290\n",
       "std          0.094818\n",
       "min          0.244385\n",
       "25%          0.389404\n",
       "50%          0.461792\n",
       "75%          0.548157\n",
       "max          0.744385\n",
       "Name: mut_gc, dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Non-SNV events by net length change:\n",
      "  Insertions      (len_diff>0) : 86,857\n",
      "  Deletions       (len_diff<0) : 169,730\n",
      "  Balanced Delins (len_diff=0) : 10,750\n"
     ]
    }
   ],
   "source": [
    "# ─── Basic cohort statistics ─────────────────────────────────\n",
    "\n",
    "print(f\"Total variants: {len(df):,}\\n\")\n",
    "\n",
    "# Variant type\n",
    "print(\"Variant type counts:\")\n",
    "display(df['variant_type'].value_counts())\n",
    "\n",
    "# Pathogenicity\n",
    "print(\"\\nPathogenicity counts:\")\n",
    "display(df['pathogenicity'].value_counts())\n",
    "\n",
    "# Top diseases\n",
    "print(\"\\nTop 10 disease names:\")\n",
    "display(df['disease_name']\n",
    "        .replace('', 'Unknown')             # collapse blanks\n",
    "        .value_counts()\n",
    "        .head(10))\n",
    "\n",
    "# ─── Indel vs. SNP breakdown ────────────────────────────────\n",
    "\n",
    "# parse ref/alt lengths\n",
    "ref_alt = df['mutation_instruction'].str.split('>', expand=True)\n",
    "df['ref_len'] = ref_alt[0].str.len().astype(int)\n",
    "df['alt_len'] = ref_alt[1].str.len().astype(int)\n",
    "df['len_diff'] = df['alt_len'] - df['ref_len']\n",
    "\n",
    "print(\"\\nLength‐difference (alt − ref) distribution:\")\n",
    "display(df['len_diff']\n",
    "        .value_counts()\n",
    "        .sort_index())\n",
    "\n",
    "# ─── Transition / transversion in SNVs ─────────────────────\n",
    "\n",
    "# only look at true SNVs (ref_len==alt_len==1)\n",
    "snv = df[(df['variant_type']=='SNV') & (df['len_diff']==0)].copy()\n",
    "def is_transition(instr):\n",
    "    pur = {'A','G'}\n",
    "    pyr = {'C','T'}\n",
    "    r,a = instr.split('>')\n",
    "    return (r in pur and a in pur) or (r in pyr and a in pyr)\n",
    "\n",
    "snv['is_transition'] = snv['mutation_instruction'].map(is_transition)\n",
    "t1 = snv['is_transition'].sum()\n",
    "t2 = (~snv['is_transition']).sum()\n",
    "print(f\"\\nSNVs: {len(snv):,}  →  Transitions: {t1:,}   Transversions: {t2:,}\\n\")\n",
    "\n",
    "# ─── GC‐content in windows (sampled) ─────────────────────────\n",
    "\n",
    "# sampling to speed up\n",
    "sample = df.sample(min(len(df), 10000), random_state=0)\n",
    "def gc_frac(s): return (s.count('G')+s.count('C'))/len(s)\n",
    "\n",
    "sample['orig_gc'] = sample['original_window'].map(gc_frac)\n",
    "sample['mut_gc' ] = sample['mutated_window'].map(gc_frac)\n",
    "\n",
    "print(\"Original‐window GC content (sample):\")\n",
    "display(sample['orig_gc'].describe())\n",
    "\n",
    "print(\"\\nMutated‐window GC content (sample):\")\n",
    "display(sample['mut_gc'].describe())\n",
    "\n",
    "\n",
    "# ─── Better Non-SNV event breakdown ────────────────────────────────\n",
    "\n",
    "non_snv = df[df['variant_type'] != 'SNV']\n",
    "\n",
    "# counts\n",
    "n_ins   = (non_snv['len_diff'] >  0).sum()\n",
    "n_del   = (non_snv['len_diff'] <  0).sum()\n",
    "n_bal   = ((non_snv['len_diff']==0) & (non_snv['ref_len']>1)).sum()\n",
    "\n",
    "print(\"Non-SNV events by net length change:\")\n",
    "print(f\"  Insertions      (len_diff>0) : {n_ins:,}\")\n",
    "print(f\"  Deletions       (len_diff<0) : {n_del:,}\")\n",
    "print(f\"  Balanced Delins (len_diff=0) : {n_bal:,}\")\n",
    "\n",
    "# catch any explicit VCF-style inversions (<INV>) if they exist\n",
    "n_inv = df['mutation_instruction'].str.contains('<INV>').sum()\n",
    "if n_inv:\n",
    "    print(f\"  Inversions                 : {n_inv:,}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mutation_instruction</th>\n",
       "      <th>original_window</th>\n",
       "      <th>mutated_window</th>\n",
       "      <th>pathogenicity</th>\n",
       "      <th>disease_name</th>\n",
       "      <th>variant_type</th>\n",
       "      <th>ref_len</th>\n",
       "      <th>alt_len</th>\n",
       "      <th>len_diff</th>\n",
       "      <th>abs_len_diff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>370378</th>\n",
       "      <td>A&gt;C</td>\n",
       "      <td>GAACTGAGGAGATAGTTTTTGTTTTTAATGATTGTGCTCTTTTAAC...</td>\n",
       "      <td>GAACTGAGGAGATAGTTTTTGTTTTTAATGATTGTGCTCTTTTAAC...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>Hereditary_cancer-predisposing_syndrome</td>\n",
       "      <td>SNV</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47441</th>\n",
       "      <td>C&gt;A</td>\n",
       "      <td>TCTTGCTGGTTTCAGGGGAGGAGCCCGCTGTGCCAGGCCCTCATCT...</td>\n",
       "      <td>TCTTGCTGGTTTCAGGGGAGGAGCCCGCTGTGCCAGGCCCTCATCT...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>not_specified</td>\n",
       "      <td>SNV</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2370658</th>\n",
       "      <td>C&gt;G</td>\n",
       "      <td>ACAGAAATAATGGAGTTAGAAAATCATTTAGTAGCCATCATAGTAA...</td>\n",
       "      <td>ACAGAAATAATGGAGTTAGAAAATCATTTAGTAGCCATCATAGTAA...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>DICER1-related_tumor_predisposition</td>\n",
       "      <td>SNV</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2479341</th>\n",
       "      <td>C&gt;A</td>\n",
       "      <td>TGAATGCTTTTAGTTGTATGTGTTTTACGTTCATAAAAGTAAAATC...</td>\n",
       "      <td>TGAATGCTTTTAGTTGTATGTGTTTTACGTTCATAAAAGTAAAATC...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>not_specified</td>\n",
       "      <td>SNV</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2340733</th>\n",
       "      <td>G&gt;A</td>\n",
       "      <td>TAAGTGGGGAAGGGCCTGCTTCCTGAGTCGGAGGCTGAGAGGATGG...</td>\n",
       "      <td>TAAGTGGGGAAGGGCCTGCTTCCTGAGTCGGAGGCTGAGAGGATGG...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>not_specified</td>\n",
       "      <td>SNV</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>312980</th>\n",
       "      <td>C&gt;T</td>\n",
       "      <td>GTCGGCCAGGGCCGCCGCGGGGCTACCGGGCGGGCTCGGGGCGGCG...</td>\n",
       "      <td>GTCGGCCAGGGCCGCCGCGGGGCTACCGGGCGGGCTCGGGGCGGCG...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>Intellectual_developmental_disorder_with_micro...</td>\n",
       "      <td>SNV</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1829920</th>\n",
       "      <td>T&gt;G</td>\n",
       "      <td>GAAGGGAATACAAGGAAGGAGGAAAGGGAGTGTTAGTTTGGGCTAT...</td>\n",
       "      <td>GAAGGGAATACAAGGAAGGAGGAAAGGGAGTGTTAGTTTGGGCTAT...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>Dilated_cardiomyopathy_1DD|Cardiovascular_phen...</td>\n",
       "      <td>SNV</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>315617</th>\n",
       "      <td>C&gt;T</td>\n",
       "      <td>TCCTGGTCCCAACCCCCTGCGCAGTATCTCTGGACGGGGCTAGACC...</td>\n",
       "      <td>TCCTGGTCCCAACCCCCTGCGCAGTATCTCTGGACGGGGCTAGACC...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>not_provided</td>\n",
       "      <td>SNV</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2279534</th>\n",
       "      <td>C&gt;T</td>\n",
       "      <td>TTACTTAGAAAAGCTCAACAAGTCTTTGGATATTTAGAGACTTTTT...</td>\n",
       "      <td>TTACTTAGAAAAGCTCAACAAGTCTTTGGATATTTAGAGACTTTTT...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>not_provided</td>\n",
       "      <td>SNV</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2536550</th>\n",
       "      <td>C&gt;T</td>\n",
       "      <td>GGGTGACACACCGGGAGAGGCTAGCAGTAAACAAAGGGAAAGGCGG...</td>\n",
       "      <td>GGGTGACACACCGGGAGAGGCTAGCAGTAAACAAAGGGAAAGGCGG...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>not_provided|Hereditary_cancer-predisposing_sy...</td>\n",
       "      <td>SNV</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        mutation_instruction  \\\n",
       "370378                   A>C   \n",
       "47441                    C>A   \n",
       "2370658                  C>G   \n",
       "2479341                  C>A   \n",
       "2340733                  G>A   \n",
       "312980                   C>T   \n",
       "1829920                  T>G   \n",
       "315617                   C>T   \n",
       "2279534                  C>T   \n",
       "2536550                  C>T   \n",
       "\n",
       "                                           original_window  \\\n",
       "370378   GAACTGAGGAGATAGTTTTTGTTTTTAATGATTGTGCTCTTTTAAC...   \n",
       "47441    TCTTGCTGGTTTCAGGGGAGGAGCCCGCTGTGCCAGGCCCTCATCT...   \n",
       "2370658  ACAGAAATAATGGAGTTAGAAAATCATTTAGTAGCCATCATAGTAA...   \n",
       "2479341  TGAATGCTTTTAGTTGTATGTGTTTTACGTTCATAAAAGTAAAATC...   \n",
       "2340733  TAAGTGGGGAAGGGCCTGCTTCCTGAGTCGGAGGCTGAGAGGATGG...   \n",
       "312980   GTCGGCCAGGGCCGCCGCGGGGCTACCGGGCGGGCTCGGGGCGGCG...   \n",
       "1829920  GAAGGGAATACAAGGAAGGAGGAAAGGGAGTGTTAGTTTGGGCTAT...   \n",
       "315617   TCCTGGTCCCAACCCCCTGCGCAGTATCTCTGGACGGGGCTAGACC...   \n",
       "2279534  TTACTTAGAAAAGCTCAACAAGTCTTTGGATATTTAGAGACTTTTT...   \n",
       "2536550  GGGTGACACACCGGGAGAGGCTAGCAGTAAACAAAGGGAAAGGCGG...   \n",
       "\n",
       "                                            mutated_window   pathogenicity  \\\n",
       "370378   GAACTGAGGAGATAGTTTTTGTTTTTAATGATTGTGCTCTTTTAAC...  not_pathogenic   \n",
       "47441    TCTTGCTGGTTTCAGGGGAGGAGCCCGCTGTGCCAGGCCCTCATCT...  not_pathogenic   \n",
       "2370658  ACAGAAATAATGGAGTTAGAAAATCATTTAGTAGCCATCATAGTAA...  not_pathogenic   \n",
       "2479341  TGAATGCTTTTAGTTGTATGTGTTTTACGTTCATAAAAGTAAAATC...  not_pathogenic   \n",
       "2340733  TAAGTGGGGAAGGGCCTGCTTCCTGAGTCGGAGGCTGAGAGGATGG...  not_pathogenic   \n",
       "312980   GTCGGCCAGGGCCGCCGCGGGGCTACCGGGCGGGCTCGGGGCGGCG...  not_pathogenic   \n",
       "1829920  GAAGGGAATACAAGGAAGGAGGAAAGGGAGTGTTAGTTTGGGCTAT...  not_pathogenic   \n",
       "315617   TCCTGGTCCCAACCCCCTGCGCAGTATCTCTGGACGGGGCTAGACC...  not_pathogenic   \n",
       "2279534  TTACTTAGAAAAGCTCAACAAGTCTTTGGATATTTAGAGACTTTTT...  not_pathogenic   \n",
       "2536550  GGGTGACACACCGGGAGAGGCTAGCAGTAAACAAAGGGAAAGGCGG...  not_pathogenic   \n",
       "\n",
       "                                              disease_name variant_type  \\\n",
       "370378             Hereditary_cancer-predisposing_syndrome          SNV   \n",
       "47441                                        not_specified          SNV   \n",
       "2370658                DICER1-related_tumor_predisposition          SNV   \n",
       "2479341                                      not_specified          SNV   \n",
       "2340733                                      not_specified          SNV   \n",
       "312980   Intellectual_developmental_disorder_with_micro...          SNV   \n",
       "1829920  Dilated_cardiomyopathy_1DD|Cardiovascular_phen...          SNV   \n",
       "315617                                        not_provided          SNV   \n",
       "2279534                                       not_provided          SNV   \n",
       "2536550  not_provided|Hereditary_cancer-predisposing_sy...          SNV   \n",
       "\n",
       "         ref_len  alt_len  len_diff  abs_len_diff  \n",
       "370378         1        1         0             0  \n",
       "47441          1        1         0             0  \n",
       "2370658        1        1         0             0  \n",
       "2479341        1        1         0             0  \n",
       "2340733        1        1         0             0  \n",
       "312980         1        1         0             0  \n",
       "1829920        1        1         0             0  \n",
       "315617         1        1         0             0  \n",
       "2279534        1        1         0             0  \n",
       "2536550        1        1         0             0  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "check to see which variant types from the vep vcf are not included in the fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[W::vcf_parse] Contig '1' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '2' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '3' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '4' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '5' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '6' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '7' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '8' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '9' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '10' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '11' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '12' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '13' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '14' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '15' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '16' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '17' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '18' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '19' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '20' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '21' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '22' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'X' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'Y' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'MT' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'NT_113889.1' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'NT_187633.1' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'NT_187661.1' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'NT_187693.1' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'NW_009646201.1' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '1' is not defined in the header. (Quick workaround: index the file with tabix.)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In VCF but not in FASTA: ['NT_113889.1', 'NT_187633.1', 'NT_187661.1', 'NT_187693.1', 'NW_009646201.1']\n",
      "In both VCF and FASTA: ['1', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '2', '20', '21', '22', '3', '4', '5', '6', '7', '8', '9', 'MT', 'X', 'Y']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[W::vcf_parse] Contig '2' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '3' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '4' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '5' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '6' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '7' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '8' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '9' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '10' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '11' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '12' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '13' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '14' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '15' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '16' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '17' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '18' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '19' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '20' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '21' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig '22' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'X' is not defined in the header. (Quick workaround: index the file with tabix.)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Counts of variants on missing contigs:\n",
      "  NT_113889.1: 1\n",
      "  NT_187633.1: 10\n",
      "  NT_187661.1: 8\n",
      "  NT_187693.1: 10\n",
      "  NW_009646201.1: 1\n",
      "\n",
      "Total variants on contigs present in both VCF and FASTA: 3494465\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[W::vcf_parse] Contig 'Y' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'MT' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'NT_113889.1' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'NT_187633.1' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'NT_187661.1' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'NT_187693.1' is not defined in the header. (Quick workaround: index the file with tabix.)\n",
      "[W::vcf_parse] Contig 'NW_009646201.1' is not defined in the header. (Quick workaround: index the file with tabix.)\n"
     ]
    }
   ],
   "source": [
    "import pysam\n",
    "from collections import Counter\n",
    "\n",
    "vcf_path   = \"SCRATCH_DIR/DNASNVData113/clinvar_data/clinvar_coding_only.vcf\"\n",
    "fasta_path = \"SCRATCH_DIR/DNASNVData113/clinvar_data/vep-cache-113/homo_sapiens/113_GRCh38/Homo_sapiens.GRCh38.dna.toplevel.fa\"\n",
    "\n",
    "# 1) open VCF\n",
    "vcf = pysam.VariantFile(vcf_path)\n",
    "\n",
    "# 2) get contigs from header if present, else from records\n",
    "vcf_contigs = set(vcf.header.contigs)\n",
    "if not vcf_contigs:\n",
    "    vcf_contigs = { rec.contig for rec in vcf }\n",
    "    vcf = pysam.VariantFile(vcf_path)  # reopen to iterate again\n",
    "\n",
    "# 3) open FASTA and get its contigs\n",
    "fa = pysam.FastaFile(fasta_path)\n",
    "fasta_contigs = set(fa.references)\n",
    "\n",
    "# 4) compute sets\n",
    "missing = sorted(vcf_contigs - fasta_contigs)\n",
    "common  = sorted(vcf_contigs & fasta_contigs)\n",
    "\n",
    "print(\"In VCF but not in FASTA:\", missing)\n",
    "print(\"In both VCF and FASTA:\", common)\n",
    "\n",
    "# 5) count variants by category\n",
    "counts_missing = Counter()\n",
    "counts_common  = 0\n",
    "\n",
    "for rec in vcf:\n",
    "    chrom = rec.contig\n",
    "    if chrom in missing:\n",
    "        counts_missing[chrom] += 1\n",
    "    elif chrom in fasta_contigs:\n",
    "        counts_common += 1\n",
    "\n",
    "# 6) report\n",
    "print(\"\\nCounts of variants on missing contigs:\")\n",
    "for contig in missing:\n",
    "    print(f\"  {contig}: {counts_missing[contig]}\")\n",
    "\n",
    "print(f\"\\nTotal variants on contigs present in both VCF and FASTA: {counts_common}\")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
